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Preface to the Third Edition 


While the first edition of this textbook was based on a one-year course in 
computational physics with a rather limited scope, its extent has been increased 
substantially in the third edition, offering the possibility to select from a broader 
range of computer experiments and to deepen the understanding of the important 
numerical methods. The computer experiments have always been a central part of 
my concepts for this book. Since Java applets, which are very convenient otherwise, 
have become more or less deprecated and their usage in a browser is no longer 
recommended for security issues, I decided to use standalone Java programs instead 
and to rewrite all of the old examples. These can also been edited and compiled 
with the “netbeans” environment and offer the same possibilities to generate a 
graphical user interface in short time. 

The major changes in the third edition are as follows. 

In the first part, a new chapter is devoted to the time-frequency analysis of 
experimental data. While the classical Fourier transform allows the calculation 
of the spectrum of a stationary signal, it is not so useful for nonstationary signals 
with significant variation of the momentaneous frequency distribution. Application 
of the Fourier transformation to short time windows, a method which is known as 
short-time Fourier transformation (STFT), allows analyzing the frequency content 
of a signal as a function of time. Good time resolution, of course, always comes 
together with a loss in frequency resolution (this is well known as “uncertainty 
principle”). The STFT method uses the same window for the whole spectrum, 
therefore the absolute time and frequency resolution is the same for low- and 
high-frequency components and the time resolution is limited by the period of the 
lowest frequencies of interest. Analysis of a signal with wavelets, on the other hand, 
uses shorter windows for the higher frequencies and keeps the relative frequency 
resolution constant while increasing the time resolution of the high-frequency 
components. The continuous wavelet transform can be very time consuming since it 
involves a convolution integral and is highly redundant. The discrete wavelet 


vii 



Preface to the Third Edition 


viii 

transform uses a finite number of orthogonal basis function and can be performed 
much faster by calculating scalar products. It is closely related to multiresolution 
analysis which analyzes a signal in terms of a basic approximation and details 
of increasing resolution. Such methods are very popular in signal processing, 
especially of audio and image data but also in medical physics and seismology. The 
principles of the construction of orthogonal wavelet families are explained in detail, 
but without too many mathematical proofs. Several popular kinds of wavelets are 
discussed, like those by Haar, Meyer and Daubechies and their application is 
explored in a series of computer experiments. 

In the second part, two new chapters have been added. First I included a dis¬ 
cussion of the advection equation. Several methods to solve the one-dimensional 
problem are discussed from very simple straightforward differencing to quite 
sophisticated Galerkin-Taylor methods. The properties of these methods are 
demonstrated in computer experiments, as well by programs in the problems section 
as by numerous figures in the text. The extension to more dimensions by finite 
volume methods and dimensional splitting are discussed. A profound understanding 
of the advection equation and its numerical solution is also the basis for the more 
complex convection and Navier-Stokes equations. 

Another chapter was added to the application of variational methods for quan¬ 
tum systems. The variational principle is very useful to calculate the groundstate 
energy. Two different types of computer experiments are performed. First we use 
the variational quantum Monte Carlo method (VQMC) for small atomic and 
molecular systems like the Helium atom and the Hydrogen molecule. We use trial 
functions which treat electron correlation explicitly by introducing a Jastrow factor 
which depends on the electron-electron distances. Such trial functions lead to 
nonseparable multidimensional integrals which can be efficiently calculated with 
the VQMC method. A second series of computer experiments studies 
exciton-phonon coupling in molecular aggregates which are of large interest for 
energy transfer in artificial and biological systems. The non-Born-Oppenheimer 
character of the wavefunction makes it necessary to optimize a large number of 
parameters. Different kinds of trial functions are applied to aggregates of up to 
100 molecules to study the localization of the lowest state (so called 
“self-trapping”). 

Apart from these newly added chapters, further improvements have been made 
throughout the book. The chapter on random numbers now discusses in more detail 
the principles of modern random number generators, especially the xorshift, mul¬ 
tiply with carry (MWC) and complementary multiply with carry (CMWC) methods. 
Nonstationary iterative Krylov-space methods for systems of linear equations are 
discussed systematically with a focus on the conjugate gradients (CG) and general 
minimum residual (GMRES) methods. The QR method for eigenvalue problems is 
now discussed in much more detail together with its connection to the power 
iteration method and the Krylov-space methods by Arnoldi and Lanczos. 
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Finally, I included a computer experiment simulating the transition between two 
states with wave packet dynamics, which is very helpful to understand the semi- 
classical approximation, especially the Landau-Zener model, which is the subject 
of another computer experiment. 

Garching, Germany Philipp O.J. Scherer 

March 2017 



Preface to the Second Edition 


This textbook introduces the main principles of computational physics, which 
include numerical methods and their application to the simulation of physical 
systems. The first edition was based on a one-year course in computational physics 
where I presented a selection of only the most important methods and applications. 
Approximately one-third of this edition is new. 1 tried to give a larger overview 
of the numerical methods, traditional ones as well as more recent developments. In 
many cases it is not possible to pin down the “best” algorithm, since this may 
depend on subtle features of a certain application, the general opinion changes from 
time to time with new methods appearing and computer architectures evolving, and 
each author is convinced that his method is the best one. Therefore I concentrated 
on a discussion of the prevalent methods and a comparison for selected examples. 
For a comprehensive description I wotdd like to refer the reader to specialized 
textbooks like “Numerical Recipes” or elementary books in the field of the engi¬ 
neering sciences. 

The major changes are as follows. 

A new chapter is dedicated to the discretization of differential equations and the 
general treatment of boundary value problems. While finite differences are a natural 
way to discretize differential operators, finite volume methods are more flexible if 
material properties like the dielectric constant are discontinuous. Both can be seen 
as special cases of the finite element methods which are omnipresent in the engi¬ 
neering sciences. The method of weighted residuals is a very general way to find the 
“best” approximation to the solution within a limited space of trial functions. It is 
relevant for finite element and finite volume methods but also for spectral methods 
which use global trial functions like polynomials or Fourier series. 

Traditionally, polynomials and splines are very often used for interpolation. 
I included a section on rational interpolation which is useful to interpolate functions 
with poles but can also be an alternative to spline interpolation due to the recent 
development of barycentric rational interpolants without poles. 

The chapter on numerical integration now discusses Clenshaw-Curtis 
and Gaussian methods in much more detail, which are important for practical 
applications due to their high accuracy. 
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Besides the elementary root finding methods like bisection and Newton- 
Raphson, also the combined methods by Dekker and Brent and a recent extension 
by Chandrapatla are discussed in detail. These methods are recommended in most 
text books. Function minimization is now discussed also with derivative free 
methods, including Brent’s golden section search method. Quasi-Newton methods 
for root finding and function minimizing are thoroughly explained. 

Eigenvalue problems are ubiquitous in physics. The QL-method, which is very 
popular for not too large matrices is included as well as analytic expressions for 
several differentiation matrices. 

The discussion of Singular value decomposition was extended and its applica¬ 
tion to low rank matrix approximation and linear fitting is discussed. 

For the integration of equations of motion (i.e. of initial value problems) many 
methods are available, often specialized for certain applications. For completeness, 
I included the predictor-corrector methods by Nordsieck and Gear which have been 
often used for molecular dynamics and the backward differentiation methods for 
stiff problems. 

A new chapter is devoted to molecular mechanics, since this is a very important 
branch of current computational physics. Typical force field terms are discussed as 
well as the calculation of gradients which are necessary for molecular dynamics 
simulations. 

The simulation of waves now includes three additional two-variable methods 
which are often used in the literature and are based on generally applicable schemes 
(leapfrog, Lax-Wendroff, Crank-Nicolson). 

The chapter on simple quantum systems was rewritten. Wave packet simulation 
has become very important in theoretical physics and theoretical chemistry. Several 
methods are compared for spatial discretization and time integration of the 
one-dimensional Schroedinger equation. The dissipative two-level system is used to 
discuss elementary operations on a Qubit. 

The book is accompanied by many computer experiments. For those readers 
who are unable to try them out, the essential results are shown by numerous figures. 

This book is intended to give the reader a good overview over the fundamental 
numerical methods and their application to a wide range of physical phenomena. 
Each chapter now starts with a small abstract, sometimes followed by necessary 
physical background information. Many references, original work as well as spe¬ 
cialized text books, are helpful for more deepened studies. 


Garching, Germany 
February 2013 


Philipp O.J. Scherer 



Preface to the First Edition 


Computers have become an integral part of modern physics. They help to acquire, 
store and process enormous amounts of experimental data. Algebra programs have 
become very powerful and give the physician the knowledge of many mathe¬ 
maticians at hand. Traditionally physics has been divided into experimental physics 
which observes phenomena occurring in the real world and theoretical physics 
which uses mathematical methods and simplified models to explain the experi¬ 
mental findings and to make predictions for future experiments. But there is also a 
new part of physics which has an ever growing importance. Computational physics 
combines the methods of the experimentalist and the theoretician. Computer sim¬ 
ulation of physical systems helps to develop models and to investigate their 
properties. 


Computers in Physics 

Experimental Physics 

data collection, storage ana processing 



Theoretical Physics Computational Physics 

approximative solutions Computer models & experiments 


This book is a compilation of the contents of a two-part course on computational 
physics which I have given at the TUM (Technische Universitat Miinchen) for 
several years on a regular basis. It attempts to give the undergraduate physics 
students a profound background in numerical methods and in computer simulation 
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methods but is also very welcome by students of mathematics and computational 
science who want to learn about applications of numerical methods in physics. This 
book may also support lecturers of computational physics and bio-computing. It 
tries to bridge between simple examples which can be solved analytically and more 
complicated but instructive applications which provide insight into the underlying 
physics by doing computer experiments. 

The first part gives an introduction into the essential methods of numerical 
mathematics which are needed for applications in physics. Basic algorithms are 
explained in detail together with limitations due to numerical inaccuracies. 
Mathematical explanations are supplemented by numerous numerical experiments. 

The second part of the book shows the application of computer simulation 
methods for a variety of physical systems with a certain focus on molecular bio¬ 
physics. The main object is the time evolution of a physical system. Starting from a 
simple rigid rotor or a mass point in a central field, important concepts of classical 
molecular dynamics are discussed. Further chapters deal with partial differential 
equations, especially the Poisson-Boltzmann equation, the diffusion equation, 
nonlinear dynamic systems and the simulation of waves on a 1-dimensional string. 
In the last chapters simple quantum systems are studied to understand e.g. expo¬ 
nential decay processes or electronic transitions during an atomic collision. 
A two-state quantum system is studied in large detail, including relaxation pro¬ 
cesses and excitation by an external field. Elementary operations on a quantum bit 
(Qubit) are simulated. 

Basic equations are derived in detail and efficient implications are discussed 
together with numerical accuracy and stability of the algorithms. Analytical results 
are given for simple test cases which serve as a benchmark for the numerical 
methods. Many computer experiments are provided realized as Java applets which 
can be run in the web browser. For a deeper insight the source code can be studied 
and modified with the free “netbeans” 1 environment. 

Garching, Germany Philipp O.J. Scherer 

April 2010 


1 www.netbeans.org. 
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Part I 

Numerical Methods 


Chapter 1 

Error Analysis 


Several sources of errors are important for numerical data processing: 

Experimental uncertainty: Input data from an experiment have a limited precision. 
Instead of the vector of exact values x the calculation uses x + Z\x, with an uncertainty 
Ax. This can lead to large uncertainties of the calculated results if an unstable 
algorithm is used or if the unavoidable error inherent to the problem is large. 

Rounding errors: The arithmetic unit of a computer uses only a subset of the real 
numbers, the so called machine numbers A C 91. The input data as well as the 
results of elementary operations have to be represented by machine numbers whereby 
rounding errors can be generated. This kind of numerical error can be avoided in 
principle by using arbitrary precision arithmetics 1 or symbolic algebra programs. 
But this is unpractical in many cases due to the increase in computing time and 
memory requirements. 

Truncation errors: Results from more complex operations like square roots or 
trigonometric functions can have even larger errors since series expansions have 
to be truncated and iterations can accumulate the errors of the individual steps. 


1.1 Machine Numbers and Rounding Errors 

Floating point numbers are internally stored as the product of sign, mantissa and a 
power of 2. According to the IEEE754 standard [1] single, double and quadruple 
precision numbers are stored as 32, 64 or 128 bits (Table 1.1). 

The sign bit s is 0 for positive and 1 for negative numbers. The exponent b is 
biased by adding E which is half of its maximum possible value (Table 1.2). 2 The 
value of a number is given by 


1 For instance the open source GNU MP bignum library. 

2 In the following the usual hexadecimal notation is used which represents a group of 4 bits by one 
of the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F. 
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Table 1.1 Binary floating-point formats 


Format 

Sign 

Exponent 

Flidden bit 

Fraction 

Precision sm 

Float 

s 

b 0 ■ 

■bj 

1 

ao ■ 

■022 

2~ u = 5.96E' 8 

Double 

s 

bo ■ 

■b\o 

1 

a 0 ■ 

•«51 

2 -53 = 1.1 IE -16 

Quadruple 

s 

bo ■ 

■bu 

1 

ao ■ 

' Alii 

2 -113 = 9.63E -35 


Table 1.2 Exponent bias E 


Decimal value 

Binary value 

Flexadecimal value 

Data type 

127 10 

IIIIIII 2 

$ 3F 

Single 

1023io 

IIIIIIIIII 2 

$ 3FF 

Double 

16383io 

IIIIIIIIIIIIII 2 

S3FFF 

Quadruple 


Table 1.3 Special double precision numbers 


Flexadecimal value 

Symbolic value 

$ 000 0000000000000 

+0 

$ 080 00000000000000 

-0 

$ 7FF 0000000000000 

+inf 

$ FFF 0000000000000 

-inf 

$ 7FF 0000000000001 ■ ■ ■ $ 7FF FFFFFFFFFFFFF 

NAN 

$ 001 0000000000000 

Min Normal 

$ 7FE FFFFFFFFFFFFF 

Max_Normal 

$ 000 0000000000001 

Min_Subnormal 

$ 000 FFFFFFFFFFFFF 

Max Subnormal 


x = (-) s xax 2 b ~ E . (1.1) 

The mantissa a is normalized such that its first bit is 1 and its value is between 1 and 2 


1.000 2 •••()<« < 1.111- -12 < IO.O2 =2 10 . (1.2) 

Since the first bit of a normalized floating point number always is 1, it is not 
necessary to store it explicitly (hidden bit or J-bit). However, since not all numbers 
can be normalized, only the range of exponents from $001 • • ■ $7FE is used for 
normalized numbers. An exponent of $000 signals that the number is not normalized 
(zero is an important example, there exist even two zero numbers with different sign) 
whereas the exponent $7FF is reserved for infinite or undefined results (Table 1.3). 
The range of normalized double precision numbers is between 

Min_Normal = 2.2250738585072014 x 1(T 308 
































1.1 Machine Numbers and Rounding Errors 


5 


and 


Max_Normal = 1.7976931348623157E x 10 308 . 


Example 

Consider the following bit pattern which represents a double precision number: 

$4059000000000000. 

The exponent is 100 0000 OIOI 2 — 011 1111 IIII 2 = 1IO 2 and the mantissa includ¬ 
ing the J-bit is 1 1001 0000 0000 • ■ - 2 . Hence the decimal value is 

1.5625 x 2 6 = 100 lo . 

Input numbers which are not machine numbers have to be rounded to the nearest 
machine number. This is formally described by a mapping ili —> A 

x —»• rd(x) 

with the property 3 


\x — rd(x )| < \x — g\ for all g e A. (1.3) 

For the special case that x is exactly in the middle between two successive machine 
numbers, a tie-breaking rule is necessary. The simplest rules are to round up always 
(round-half-up) or always down (round-half-down). However, these are not sym¬ 
metric and produce a bias in the average round-off error. The IEEE-754 standard 
[1] recommends the round-to-nearest-even method, i.e. the least significant bit of 
the rounded number should always be zero. Alternatives are round-to-nearest-odd, 
stochastic rounding and alternating rounding. 

The cases of exponent overflow and exponent underflow need special attention: 

Whenever the exponent b has the maximum possible value b — /? nlax and a = 
1.11 • - • 11 has to be rounded to a' = 10.00 ■ • ■ 0, the rounded number is not a 
machine number and the result is ± inf. 

Numbers in the range 2 fcmin > |x| > 2 hmin ~ t can be represented with loss of 
accuracy by denormalized machine numbers. Their mantissa cannot be normalized 
since it is a < 1 and the exponent has the smallest possible value b = b n]m . Even 
smaller numbers with \x\ < 2~' +hmin have to be rounded to ±0. 


3 Sometimes rounding is replaced by a simpler truncation operation which, however leads to signif¬ 
icantly larger rounding errors. 
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1.00x2 _1 1.00 1.01 1.10 1.11 1.00x2 1 

Fig. 1.1 (Round to nearest) Normalized machine numbers with t = 3 binary digits are shown. 
Rounding to the nearest machine number produces a round-off error which is bounded by half the 
spacing of the machine numbers 


The maximum rounding error for normalized numbers with t binary digits 
a' — s x 2 h ~ E x l.a\d 2 ■ ■ ■ a,-\ 
is given by (Fig. 1.1) 

\a — a'\ < 2 b ~ E x 2~' 
and the relative error is bounded by 
rd(x ) — x 


2~' x 2 b 

< -r- < 2 ~'. 


\a\ x 2 b 

The error bound determines the relative machine precision 4 
£m = 2 ' 


(1.4) 


(1.5) 


( 1 . 6 ) 


(1.7) 


and the rounding operation can be described by 

rd(x) = *(1 + e) with |e| < e M . (1.8) 

The round-off error takes its maximum value if the mantissa is close to 1. Consider 
a number 


X = l + £. 

If e < £m then rd(x) = 1 whereas for £ > £m rounding gives rd(x) = 1 + 2 1-f 
(Fig. 1.2). Hence £m is given by the largest number e for which rd(1.0 + e) — 1.0 
and is therefore also called unit roundoff. 


4 A1so known as machine epsilon. 
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Fig. 1.2 (Unit round off) 


| 1.000000 

■0 

0.000000 

■0 


1.000000 ■ 

■0 


lo.oooooo ■ 

■0 


11.000000 ■ 

■o 


lo.oooooo ■ 

■o 


01111111 


10000000 


1.000000 -0 


1.000000 -0 


1.000000 -1 


10000001 


1.2 Numerical Errors of Elementary Floating Point 
Operations 

Even for two machine numbers x, y e A the results of addition, subtraction, multi¬ 
plication or division are not necessarily machine numbers. We have to expect some 
additional round-off errors from all these elementary operations [2], We assume 
that the results of elementary operations are approximated by machine numbers as 
precisely as possible. The IEEE754 standard [1] requires that the exact operations 
x + y, x — y, x x y, x y are approximated by floating point operations A —> A 
with the property: 

fl+(x, y) = rd (x + y) 
fl-(x, y) = rd(x - y ) 
fl*(x, y) = rd(x x y) 

fl+(x, y) = rd(x -E y). (1.9) 


1.2.1 Numerical Extinction 

For an addition or subtraction one summand has to be denormalized to line up the 
exponents (for simplicity we consider only the case x > 0, y > 0) 

x + y = a x 2 b *~ E + a, 2 by ~ E = (a x + a y 2 b y- b *)2 b *~ E . (1.10) 

If the two numbers differ much in their magnitude, numerical extinction can happen. 
Consider the following case: 

y < 2 bx ~ E x 2~' 
a y 2 by ~ bx < 2~'. 


( 1 . 11 ) 
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The mantissa of the exact sum is 

a x + a y 2 by ~ bx = l.a 2 ■ ■ ■ a,_i01/3 2 • ■ • 0 t -i- (1.12) 

Rounding to the nearest machine number gives 

rd(x + y) — 2 bx x (l.a 2 ■ ■ ■ a,_i) = x (1-13) 

since 

|0.01/3 2 • • • Pt-i -0| < 10.011 •• • 1| = 0.1 -0.00 •••01 

|0.01/? 2 ■ • ■ 0t~l - 1| > 10.01 - 1| =0.11. (1.14) 

Consider now the case 

y < x x 2~'~ l = a x x 2 hx ~ E ~ , ~ l < 2 bx ~ E ~ t . (1.15) 

For normalized numbers the mantissa is in the interval 

l<\a x \<2 (1.16) 

hence we have 

rd(x + y) = x if — < 2 _,_1 = —. (1.17) 

x 2 

Especially for x = 1 we have 

rd(l+y) = 1 if y < 2~‘ = 0.00 ••• 0 f _i 1,000 •• • (1.18) 


2 ' could be rounded to 0 or to 2 1 ' since the distance is the same |2 * — 0| = 
\2~' -2 l ~‘\ =2~'. 

The smallest machine number with fl+{ 1, c) > 1 is either e = 0.00 • • • 1,0 • • • = 
2~' or £ = 0.00 ■ • ■ 1,0 • • • 01 2 ,-i = 2 _, (1 + 2 1_f )- Hence the machine precision 
£m can be determined by looking for the smallest (positive) machine number £ for 
which // + (1, £) > 1. 

1.2.2 Addition 

Consider the sum of two floating point numbers 


y = xi + x 2 . 


(1-19) 
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First the input data have to be approximated by machine numbers: 
x\ —> rd{x i) = x\ (1 + £i) 

X2 -> rd(x 2) = *2(1 + £ 2 ) (1-20) 

The addition of the two summands may produce another error a since the result has 
to be rounded. The numerical result is 


y = fl+(rd(xi),rd(x 2 )) = (xi(l + £\) +x 2 (l + er 2 ))(l +«)■ 
Neglecting higher orders of the error terms we have in first order 

y = X\ +X2+ X\£i + x 2 £l + (.Vl + X 2 )ol 


and the relative error of the numerical sum is 


.V ~ y 

y 


X\ X 2 

-£l H- £2 + OL. 

X\ + X2 Xi + X 2 


( 1 . 21 ) 


( 1 . 22 ) 


(1.23) 


If x\ ~ —A '2 then numerical extinction can produce large relative errors and uncer¬ 
tainties of the input data can be strongly enhanced. 


1.2.3 Multiplication 

Consider the multiplication of two floating point numbers 

y — X\ x x 2 . (1.24) 

The numerical result is 


y = fl*(rd(xi),rd(x 2 )) = *i(l +£i)^ 2(1 +£ 2)(1 +p) & ^ 1 ^ 2(1 +£1 +£2 + n) 

(1.25) 


with the relative error 




The relative errors of the input data and of the multiplication just add up to the total 
relative error. There is no enhancement. Similarly for a division 


x\ 

y = — 

X 2 


(1.27) 
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the relative error is 


.V - y 

y 


1 + £i — £2 + /r. 


(1.28) 


1.3 Error Propagation 


Consider an algorithm consisting of a sequence of elementary operations. From the 
set of input data which is denoted by the vector 

x = (xi---x„) (1-29) 

a set of output data is calculated 

y = (yi •■•?«). (1.30) 


Formally this can be denoted by a vector function 

y = <p(x) 


(1.31) 


which can be written as a product of r simpler functions representing the elementary 
operations 


(p = ip ^ r ' 1 


x cp ( '' " • ■ - <pV. 


(1.32) 


Starting with x intermediate results x, = (x,- |, • • ■ x;, h ) are calculated until the output 
data y result from the last step: 

xi = </ n (x) 

X2 = <P (2) (X!) 


Xr-t = <P (r 1 ) (X,._ 2 ) 

y = ip (r) (x r _i). (1.33) 

In the following we analyze the influence of numerical errors onto the final results. 
We treat all errors as small quantities and neglect higher orders. Due to round-off 
errors and possible experimental uncertainties the input data are not exactly given 
by x but by 


x + Ax. 


(1.34) 
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The first step of the algorithm produces the result 
Xi — rd(ip (1> (x + Zlx)). 

Taylor series expansion gives in first order 

x~i = (ip (1) (x) + fl/Mx) (1 + E i) H- 

with the partial derivatives 



and the round-off errors of the first step 



The error of the first intermediate result is 


Ax\ — xi — Xi = Dip^Ax + ip < - 1 \x)E i. 


The second intermediate result is 

x ~2 = (^ (2) (x"i)) (1 + E 2 ) = <^ ( 2 ) (X! + AX\) (1 + E 2 ) 
= x 2 (l + E 2 ) + Dip a) D^ l) Ax + Dip (2) x l E l 
with the error 

Ax 2 = x 2 E 2 + Dip^ Ax + Dip (1) x\E\. 

Finally the error of the result is 


(1.35) 


(1.36) 


(1.37) 


(1.38) 


(1.39) 


(1.40) 


(1.41) 


Ay = y E r + D<p (r) ■ ■ ■ Dtp w Ax + Dip (r) • ■ • Dip (2) x x E x H- D^ r) x r _ x E r _ x . 

(1.42) 

The product of the matrices Dip (n ■ ■ ■ D<p {i) is the matrix which contains the deriv¬ 
atives of the output data with respect to the input data (chain rule) 
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/By i Byi\ 

dxi ' ' ' fir.. 


Dp = Dp^ ■ 


Dp' 


CD _ 


dx„ 


dy m dy m 

\ dx dx„ ) 


(1.43) 


The first two contributions to the total error do not depend on the way in which the 
algorithm is divided into elementary steps in contrary to the remaining summands. 
Hence the inevitable error which is inherent to the problem can be estimated as [2] 


Z\ (ln) y,. = e M \ yj \ + 

j =1 


dy t 

dxj 


\Axj\ 


(1.44) 


or in case the error of the input data is dominated by the round-off errors | Axj \ < 

£m\xj\ 


A^yi = e M \yi\ + £m 

j =i 


dyt 


dxj 


(1.45) 


Additional errors which are smaller than this inevitable error can be regarded as 
harmless. If all errors are harmless, the algorithm can be considered well behaved. 


1.4 Stability of Iterative Algorithms 

Often iterative algorithms are used which generate successive values starting from 
an initial value xq according to an iteration method 


x j+l = (1.46) 

for instance to solve a large system of equations or to approximate a time evolution 
Xj ~ x( j At). Consider first a linear iteration equation which can be written in matrix 
form as 


x j+ 1 = Axj. (1.47) 

If the matrix A is the same for all steps we have simply 

Xj = A j x 0 . (1.48) 

Consider the unavoidable error originating from errors Ax of the start values: 

Xj = A j (x o + Ax) = A j x o + A-i Ax. 


(1.49) 
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The initial errors Ax can be enhanced exponentially if A has at least one eigenvalue 5 
A with |A| > 1. On the other hand the algorithm is conditionally stable if for all 
eigenvalues |A| < 1 holds. For a more general nonlinear iteration 

Xj+t =<p(xj) (1.50) 

the error propagates according to 

X! = <p(x 0 ) + Dip Ax 

x 2 = ip{xi) = ip(ip(x 0 )) + (Dip) 2 Ax 

xj =ip(ip---ip(x 0 )) + (Dp>) J Ax. (1.51) 

The algorithm is conditionally stable if all eigenvalues of the derivative matrix Dip 
have absolute values |A| < 1. 

1.5 Example: Rotation 

Consider a simple rotation in the complex plane. The equation of motion 


z = iuz (1-52) 

obviously has the exact solution 

z(f)=z 0 e'“ r . (1.53) 

As a simple algorithm for numerical integration we use a time grid 

t, = j At j= 0,1,2-■■ (1.54) 

Zj=z(tj) (1.55) 

and iterate the function values 

Zj+ 1 = Zj + z(tj) = (1 + iuiAt)zj. (1.56) 

Since 

|1 +icoAt\ = > 1 (1.57) 


5 The eigenvalues of A are solutions of the eigenvalue equation Ax = Ax (Chap. 10). 
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uncertainties in the initial condition will grow exponentially and the algorithm is not 
stable. A stable method is obtained by taking the derivative in the middle of the time 
interval (p. 296) 

and making the approximation (p. 297) 

/ At\ z(t) + z(t + At) 

T + Tr—2—■ 

This gives the implicit equation 

, ■ . Z/+1 + Zj 

Zj +1 — Zj + i u> At -—- 

which can be solved by 

1 iujAt 
— 1 ' 2 
— , _ iuAt 

1 2 

Now we have 


1 + 

1 

+ 

e to 

i iujAt 

1 2 



= 1 


and the calculated orbit is stable. 


1.6 Truncation Error 


(1.58) 


(1.59) 


(1.60) 


The algorithm in the last example is stable but of course not perfect. Each step 
produces an error due to the finite time step. The exact solution 


z(t + At) = z(t)e ,ulA ' = z(t)(l + iuiAt - 
is approximated by 


co 2 At 2 


-icu 3 At 3 


(1.61) 


z(t + At) w Z (t) 


1 + 


1 - 


icoAt 
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= z(t) 


( iujAt\( iivAt 
( 1 + — )( 1 + — 


u>At urAt 2 iu? At 3 


8 


= z(*)f 


uj 2 At 2 —iuj 3 At 3 
1 ~\~ iujAt —-“I - - 


(1.62) 


(1.63) 


which deviates from the exact solution by a term of the order 0(At 3 ), hence the 
local error order of this algorithm is 0(At 3 ) which is indicated by writing 


1 I iujAt 

z(t + At) = Z(t) + 0(At 3 ). (1.64) 

* 2 

Integration up to a total time T = NAt accumulates a global error of the order 
N At 3 = TAt 2 . 


Problems 

Problem 1.1 Machine Precision 

In this computer experiment we determine the machine precision Em- Starting with a 
value of 1.0, x is divided repeatedly by 2 until numerical addition of 1 and .v = 2~ M 
gives 1. Compare single and double precision calculations. 

Problem 1.2 Maximum and Minimum Integers 

Integers are used as counters or to encode elements of a finite set like characters or 
colors. There are different integer formats available which store signed or unsigned 
integers of different length (Table 1.4). There is no infinite integer and addition of 1 
to the maximum integer gives the minimum integer. 

In this computer experiment we determine the smallest and largest integer num¬ 
bers. Beginning with I = 1 we add repeatedly 1 until the condition I + 1 > I 
becomes invalid or subtract repeatedly 1 until I — 1 < I becomes invalid. For the 
64 bit long integer format this takes too long. Here we multiply alternatively I by 2 
until I — 1 < / becomes invalid. For the character format the corresponding ordinal 
number is shown which is obtained by casting the character to an integer. 


Table 1.4 Maximum and minimum integers 


Java format 

Bit length 

Minimum 

Maximum 

Byte 

8 

-128 

127 

Short 

16 

-32768 

32767 

Integer 

32 

-2147483647 

2147483648 

Long 

64 

-9223372036854775808 

9223372036854775807 

Char 

16 

0 

65535 
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Problem 1.3 Truncation Error 

This computer experiment approximates the cosine function by a truncated Taylor 
series 

^max ^.2 n x 2 x 4 x 6 

cos(x) sa mycos(x, n max ) = = 1 “ ^ + 24 ~720 + "' (1 ' 65) 

n= 0 

in the interval — ir/2 < x < 7r/2. The function mycos(x, n max ) is numerically 
compared to the intrinsic cosine function. 



Chapter 2 

Interpolation 


Experiments usually produce a discrete set of data points (x, , /, ) which represent 
the value of a function /(x) for a finite set of arguments {xo .. . x„}. If additional 
data points are needed, for instance to draw a continuous curve, interpolation is 
necessary. Interpolation also can be helpful to represent a complicated function by a 
simpler one or to develop more sophisticated numerical methods for the calculation 
of numerical derivatives and integrals. In the following we concentrate on the most 
important interpolating functions which are polynomials, splines and rational func¬ 
tions. Trigonometric interpolation is discussed in Chap. 7. An interpolating function 
reproduces the given function values at the interpolation points exactly (Fig. 2.1). 
The more general procedure of curve fitting, where this requirement is relaxed, is 
discussed in Chap. 11. 

The interpolating polynomial can be explicitly constructed with the Lagrange 
method. Newton’s method is numerically efficient if the polynomial has to be evalu¬ 
ated at many interpolating points and Neville’s method has advantages if the poly¬ 
nomial is not needed explicitly and has to be evaluated only at one interpolation 
point. 

Polynomials are not well suited for interpolation over a larger range. Spline 
functions can be superior which are piecewise defined polynomials. Especially cubic 
splines are often used to draw smooth curves. Curves with poles can be represented 
by rational interpolating functions whereas a special class of rational interpolants 
without poles provides a rather new alternative to spline interpolation. 


2.1 Interpolating Functions 

Consider the following problem: Given are n + 1 sample points (x,-, f) , i = ()■■■ n 
and a function of x which depends on n + 1 parameters a,: 


<t>(x; «o • • • a«). 


( 2 . 1 ) 
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Fig. 2.1 (Interpolating 
function) The interpolating 
function O(jc) reproduces a 
given data set <t> (jc; ) = /,■ 
and provides an estimate of 
the function fix) between 
the data points 



The parameters are to be determined such that the interpolating function has the 
proper values at all sample points (Fig. 2.1) 

<$>(.T/; a o • • • a„) — f-, i = 0 • • • n. (2.2) 

An interpolation problem is called linear if the interpolating function is a linear 
combination of functions 


<&(x; a 0 ■ ■ ■ a„) = a 0 <t>o(x) + a\<$>\(x) H- a„<P„(x). (2.3) 

Important examples are 

• polynomials 

a o + a\x + • ■ • a n x n (2.4) 

• trigonometric functions 

ciq T ci ] e' A -\~ a 2 ^" T • • • ci n & n ' x (2.5) 

• spline functions which are piecewise polynomials, for instance the cubic spline 

s(x) = a,- + pi (x - xP + 7 ,- (x - x,) 2 + S,- (x - x,) 3 x,- < x < x,+i. (2.6) 

Important examples for nonlinear interpolating functions are 

• rational functions 

P o + P l-vi- Pmx m 

q 0 + q\X H- q N x N 


(2.7) 
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• exponential functions 

a 0 e XoX + a ^ H-. 

where amplitudes a, and exponents A; have to be optimized. 


( 2 . 8 ) 


2.2 Polynomial Interpolation 

For n + 1 sample points (x t , f, ), i — 0- • n, x, ^ Xj there exists exactly one 
interpolating polynomial of degree n with 

p(xi) — /), i=0---n. (2.9) 

2.2.1 Lagrange Polynomials 


Lagrange polynomials [3] are defined as 

(X - X 0 ) ■ ■ ■ (x - Xf—i ) (x - x i+ i) ■ ■ ■ (x - x n ) 


L,(X) = 


( 2 . 10 ) 


(Xi - Xq) ■ ■ ■ {Xi - Xi-i)(Xi - x i+ i) ■ ■ ■ (Xi - x„) 

They are of degree n and have the property 

L t(x k ) = 5i, k . (2.11) 

The interpolating polynomial is given in terms of Lagrange polynomials by 


n n 


p(x) = ^ fiU (x) = ^ ft J~( 


1=0 


/=0 k=0,k^i 


X — X/c 
Xi - x k ‘ 


( 2 . 12 ) 


2.2.2 Barycentric Lagrange Interpolation 

With the polynomial 


u(x) = P[(x - Xi) 
i =0 


(2.13) 
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the Lagrange polynomial can be written as 
u(x) 1 

L; (x) = - 7 -r 

x ~ x i llk=0,k^( x i ~ x k) 

which, introducing the Barycentric weights [4] 

1 


rw*(* Xk ' > 

becomes the first form of the barycentric interpolation formula 

Uj 

L,(.v:) = u>(x) -. 

x — Xi 

The interpolating polynomial can now be evaluated according to 

n n 

p{x) = X! f‘ Li W = ^(x) X! fi ~~ 

X Xi 


i= 0 


i= 0 


(2.14) 


(2.15) 


(2.16) 


(2.17) 


Having computed the weights n, , evaluation of the polynomial only requires O(n) 
operations whereas calculation of all the Lagrange polynomials requires O ( n 2 ) oper¬ 
ations. Calculation of uj(x) can be avoided considering that 


Pi(x) = ^ Li(x) = u(x) ^ 

i =0 /= 0 * 

is a polynomial of degree n with 
pi(xi) = 1 i =0...n. 

But this is only possible if 


P i(x) = I- 


Therefore 


p(x) = 


P(x) 

P\(x) 



(2.18) 


(2.19) 


( 2 . 20 ) 


( 2 . 21 ) 


which is known as the second form of the barycentric interpolation formula. 
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2.2.3 Newton’s Divided Differences 


Newton’s method of divided differences [5] is an alternative for efficient numerical 
calculations [6], Rewrite 

fix) — f(xo) 

fix) = fix o) + J J ix - xo). (2.22) 

X — Xo 

With the first order divided difference 


f[x,x 0 ] 


fix) ~ fix o) 

X — Xq 


(2.23) 


this becomes 

f[x,x 0 ] = f[x i,x 0 ] + 


f[x,x 0 \-f[x i,x 0 ] 
- ix - xi) 


X — X\ 

and with the second order divided difference 

, f[x, x 0 ] - f[xu X 0 ] fix) - fix o) 
j[x,x 0 ,*iJ = - = - 

X — X\ (x—Xo)(^—Xi) 

_ fix) _^_ fix l) _^ 

ix - Jto)U- - Xl) ix 1 - x 0 )ixi - x) 


(2.24) 


_ fix\) ~ fix o) 

ix 1 - x 0 )ix - Xl) 

fix o) 

(x 0 - xi) (x 0 - x) 

(2.25) 


we have 

fix) = fix o) + (x - x 0 ) f[x !, x 0 ] + (x - x 0 )(x - xi) f[x, x 0 , xi], (2.26) 
Higher order divided differences are defined recursively by 
/[xix 2 • • • x r _i] - f[x 2 ■ ■ ■ x r _ix r ] 


f[x 1X2 • ■ ■ X r _iX r ] = 


Xl — x r 


(2.27) 


They are invariant against permutation of the arguments which can be seen from the 
explicit formula 


/[xix 2 ---x r ] = y. 


fixf) 


n ijtifxk - x ^' 


(2.28) 


Finally we have 


fix) = pix) + qix) 


(2.29) 
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with a polynomial of degree n 

P(x) = f(x o) + /[xi, x 0 ](x - x 0 ) + /[x 2 X!X 0 ](x - x 0 )(x - xO H- 


-h /[x„x„_! • • • X 0 ](x - x 0 )(x - Xl) • • • (x - X„_l) 


(2.30) 


and the function 


q(x) — f[xx n ■ ■ ■ x 0 ](x - x 0 ) • ■ • (x - x n ). (2.31) 

Obviously q{x{) = 0 , i = 0 • • • n, hence p(x ) is the interpolating polynomial. 

Algorithm 

The divided differences are arranged in the following way: 

/o 

f\ f[x oXl] 

fn — 1 f [X/i — 2.X ji — 1 ] f [X/j — 3X/1—2X/1 — 1 ] ■ ■ ■ f\x 0 ■ ■ • X n —l] 
fn f [X„ -1 X„ ] f [X„ _ 2 X„ _ 1 X n ] ■■■ f[x 1 ' ' • X„ _ 1 X„ ] / [X 0 X1 • • • X„_ 1 X„ ] 

(2.32) 

Since only the diagonal elements are needed, a one-dimensional data array f[0] ■ • • 
t[n] is sufficient for the calculation of the polynomial coefficients: 

for i:=0 to n do begin 

t[i]:=f[i]; 

for k:=i-l downto 0 do 

t [k]:=(t [k+1 ] -t [k] )/(x [i] -x [k]); 
a[i] :=t[0]; 
end; 

The value of the polynomial is then evaluated by 
p:=a[ n]; 

for i:=n-l downto 0 do 
p:=p*(x-x[i])+a[i]; 


2.2.4 Neville Method 

The Neville method [7] is advantageous if the polynomial is not needed explicitly 
and has to be evaluated only at one point. Consider the interpolating polynomial for 
the points xq • • • x*, which will be denoted as /o. i..../ ; (x). Obviously 
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^ 0 ,!,-*(*) 


(x - x 0 )P h .. k (x) - (x - X k )P 0 ... k -l(x) 
Xk - X 0 


since for x — X\ ■ • • Xk-\ the right hand side is 


(x - x 0 )/(x) - (x - x k )f(x ) 
x k - *o 

For x = xo we have 
-(x 0 - x k )f(x) 


= fix). 


X k - Xo 
and finally for x = x k 
(x k - x 0 )/(x) 


= fix ) 


Xlc - x 0 


= fix). 


Algorithm: 

We use the following scheme to calculate Pq\...„(x) recursively: 


(2.33) 


(2.34) 


(2.35) 


(2.36) 


Po 

Pi Poi 

Pi P\i Pan ( 2 . 37 ) 

Pn Pn — l,n Pn—2,n—l,n ' ' ' Poi—n 

The first column contains the function values P, (x) = /, . The value Pq\...„ can be 
calculated using a 1-dimensional data array /t[0] • • • p[n]\ 

for i:=0 to n do begin 

p[i] : =f[i]; 

for k:=i-l downto 0 do 

P [k]:=(p [k+1 ] * (x-x[ k] )-p [k ] * (x-x [i]) )/(x [k] -x [i]); 

end; 

f:=p[0]; 


2.2.5 Error of Polynomial Interpolation 

The error of polynomial interpolation [8] can be estimated with the help of the 
following theorem: 

If fix) is n + 1 times differentiable then for each x there exists £ within the 
smallest interval containing x as well as all the x,- with 
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Fig. 2.2 (Interpolating 
polynomial) The interpolated 
function (solid cun>e) and 
the interpolating polynomial 
(broken curve ) for the 
example (2.40) are compared 



q{x) = ]~[u - Xi) 


1=0 


f n+l) (0 

(n + 1)! ' 

From a discussion of the function 

n 

■’(x) = P|(r - Xi) 


(2.38) 


w( 


(2.39) 


i =o 


it can be seen that the error increases rapidly outside the region of the sample points 
(extrapolation is dangerous!). As an example consider the sample points (Fig. 2.2) 


7 r 37 t 

f(x) = sin(.t) Xi = 0, 7T, —, 2?r. 

The maximum interpolation error is estimated by(|/ (n+1) | < 1) 

I fix) - p(x) I < ^ °' 3 


(2.40) 


(2.41) 


whereas the error increases rapidly outside the interval 0 < x < 2n (Fig. 2.3). 


2.3 Spline Interpolation 


Polynomials are not well suited for interpolation over a larger range. Often spline 
functions are superior which are piecewise defined polynomials [9, 10]. The simplest 
case is a linear spline which just connects the sampling points by straight lines: 

, \ i y>+ 1 — y> / \ 

PAx) — H-(a- - x{) 


X i+ 1 - Xi 


(2.42) 
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Fig. 2.3 (Interpolation 
error) The polynomial u>(x) 
is shown for the example 
(2.40). Its roots xi are given 
by the x values of the sample 
points (circles). Inside the 
interval xq - ■ ■ X4 the absolute 
value of u is bounded by 
|cu(x)| < 35 whereas outside 
the interval it increases very 
rapidly 



s(x) = pi (x) where x,- < x < x, + i. (2.43) 

The most important case is the cubic spline which is given in the interval x,- < x < 
Xi+i by 


Pi (X ) = a,- + 0i (x - Xi ) + 7 i (x - Xi ) 2 + Si (x - Xj ) 3 . (2.44) 

We want to have a smooth interpolation and assume that the interpolating function 
and their first two derivatives are continuous. Hence we have for the inner boundaries: 

i = 0 1 


Pi (x, + l ) = Pi+ l(x,+i) 

(2.45) 

Pi(x i+ 1 ) = p'i+ i(-T-+i) 

(2.46) 

Pi(Xi + 1 ) = Pi+l(Xi+l). 

(2.47) 


We have to specify boundary conditions at Xq and x„. The most common choice are 
natural boundary conditions s"(xo) = s"(x n ) = 0, but also periodic boundary con¬ 
ditions s" (xo) = s"(x n ), s'(xo) = s' (x„), s(xq) = s(x„) or given derivative values 
s'(xo) and s'(x„) are often used. The second derivative is a linear function [2] 

p'j(x) = 27 1 + 6 ( 5 /(x - Xi) (2.48) 

which can be written using h l+ \ = x, + i — X; and M, — s" (x, ) as 
„ (x — X;) (x/4-1 — x) 

p'/(x) = M i+1 — - - + Mi - 2 i = 0.. - „ - 1 (2.49) 

"i+1 "i+1 


since 
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Pi(Xi) = Mi -^—— = s"(Xi) 
hi+ 1 

/// \ » ^ (*^/+l -^i) X 

Pi (*/+1) = M/+1---= s te+i). 

"i+l 

Integration gives with the two constants A, and Bj 

, ,, (x-Xif (x i+ i-x) 2 

Pi (x) = M i+ 1 —- Mi ——- + A; 

2ft/ + i 2rt/+i 

, x ,, (x-Xj) 3 (x i+ i-x) 3 

Pi (x) = M i+ i ——-h M, ——-h A/ (x - X,) + Bj . 

Orti+l D«/ + l 

From s(Xi) = j,- and .v(x !+ i) = y, + i we have 


/j? + i 

M,-i±i + Bj = y, 
6 


Mj+\— -1- Ajh, + i + Bj — y, + 1 

6 

and hence 

A-+i 

B, = - M,--£i 

6 

a y/+i yi hj+i .. . , 

A,- = ^-— (M;+1 - Mi) . 

h i+ 1 6 

Now the polynomial is 

M/+i ■> M/ t 

p, (x) = —-(x - X/) - —-(x - x/ - h i+ 1 ) + A, (x - Xj) + B, 

6/ti+i 6n, + i 

r i3 / V' + l Mi \ Mj 2 

= (x — X,) I-I H-3«,+i(x — X,) 

V 6/ti+i 6/;, + i / 6 h i+1 


_ (Mi^^X 

V 6A.-+1 6/;, + i / 
(X_X,) ( A, “ 6 ^ 3A?+1 ) 


Mj , \ Mj ■, 

—3/r /+1 J + B, + i+i- 


Comparison with 


Pi(x ) = a,- + /3,(x - x,) + 7,-(x - x/) 2 + 5,(x - x,) 3 


M ; 2 

“h ^ ^/+1 — yi 


( 2 . 60 ) 
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Pi = Ai - 


A / +1 M/ 
2 


yt +1 - y- 

* 1+1 


M, + i + 2M,- 

- A,+,--- 

6 



(2.61) 

(2.62) 


Mj+i - Mj 

6*, + i 


(2.63) 


Finally we calculate M, from the continuity of s'(x). Substituting for A; in /j'(x) we 
have 


,, . ,, (■ x-Xi ) 2 (x,+i-x ) 2 y+i-y/ */+i... ... 

P to = M '+i—n- M i~V, -+ —;-— (-Mi+i - M ; ) 

2*/+i 2«,+i */+1 6 


and from p' it (x,) = p-(x,-) it follows 

hi , - >7-i Ai... „ . 

M,-— +---—(M; - M+0 

2 *, 6 

= - ~ ~ V/ - ^(Mi+i - M,-) 

2 /Z;+i 6 

A,- hi A ,+1 *,+i y,+i — yj yi — y,_i 

+ M,_!-2- + Mj —+ M !+1 —■ = ^±1-U _ _i 

3 6 3 6 /z/+i /i; 

which is a system of linear equations for the M,-. Using the abbreviations 

*/+i 


A; = 


h, + *i+i 


Pi - 1 A; - 


hi 


hi + *i+i 


t/,- = 


/ jVj+i - yi _ yi - Vi- i \ 

V *,+1 A,- / 


A, + *,+i \ *,+i 

we have 

Pi Mi -1 + 2M,- + A/M, + i — dj i — 1 • • • n — 1. 
We define for natural boundary conditions 


(2.64) 


(2.65) 

( 2 . 66 ) 

(2.67) 

( 2 . 68 ) 

(2.69) 

(2.70) 


Aq = 0 p n =0 do — 0 <7,i = 0 


(2.71) 
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and in case of given derivative values 

, , , , 6 /yi -y 0 A 6 / , 

A 0 =l i-in = 1 d 0 — — I—— - y 0 J d n = — ly n 

The system of equations has the form 


2 Ao 
Hi 2 Ai 
Hi 2 A 2 

Hn — l 2 A „_1 
Hn 2 


For periodic boundary conditions we define 

\ - hl . _ , \ j _ 6 

/ v! — . . — i A, ; — 

hi+h n )n + h ri 

and the system of equations is (with M„ = Mo) 

2 Ai /ui 

Hi 2 A 2 
M 3 2 A 3 


J/i ~ y„ 
hn 




(2.72) 



" ' 


do 


Mi 


d\ 


m 2 

— 

d 2 


M n -\ 


dn ] 


_ M„ _ 


dn 


(2.73) 


(2?4) 

V /ii h n ) 


K 


Hn —1 2 A h _i 
Hn 2 



Ml 


" d 1 " 


m 2 


d2 


m 3 

— 



M n —i 


dn —1 


_ M n _ 


dn _ 


(2.75) 


All this tridiagonal systems can be easily solved with a special Gaussian elimination 
method (Sects. 5.3 and 5.4) 


2.4 Rational Interpolation 

The use of rational approximants allows to interpolate functions with poles, where 
polynomial interpolation can give poor results [2]. Rational approximants without 
poles [ 11 ] are also well suited for the case of equidistant x,, where higher order 
polynomials tend to become unstable. The main disadvantages are additional poles 
which are difficult to control and the appearance of unattainable points. Recent 
developments using the barycentric form of the interpolating function [11-13] helped 
to overcome these difficulties. 




















2.4 Rational Interpolation 


29 


2.4.1 Pade Approximant 

ThePade approximant [14] of order [M/N] to a function fix) is the rational function 


Po + p\x + ... p M x M 
q 0 + q\X + ... q N x N 



(2.76) 


which reproduces the McLaurin series (the Taylor series at x = 0) of 


f(x ) = ao + ci\x + a 2 x 2 + ... 


(2.77) 


up to order M + N, i.e. 


/(0) = R( 0) 



d (M+A0 d (M+iV) 


dx ( M+N) f } d y (M+N) R ^' 


(2.78) 


Multiplication gives 

po + p\x H-h Pmx M — (qo + q\x H-h q N x N ){ao + a\x + ...) ( 2 . 79 ) 

and collecting powers of x we find the system of equations 

Po — qoao 

Pi = qoai + qiCio 

P 2 = qoa 2 + a\q\ + a 0 q 2 

Pm = qociM + a«-i?i + • • • + aoqM 
0 = qoaM+i + qiOM + ■ ■ ■ + qN^M-N+i 

0 = qoOM+N + q\OM+N-\ + • • • + qN^M ( 2 . 80 ) 


where 


a„ — 0 for n < 0 


(2.81) 


qj — 0 for j > N. 


(2.82) 
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Example: Calculate the [3, 3] approximant to tan(x). 

The Laurent series of the tangent is 

1 , 2 s 

tan(x) = x + -x 3 + —x 5 +_ (2.83) 

We set <70 = 1 • Comparison of the coefficients of the polynomial 


Po + p\x + pox 2 + P 3 X 3 = (1 + q\x + q 2 X 2 + qsx 3 ) ^x + -x 3 + —x 5 ^ 

(2.84) 

gives the equations 

x° : po = 0 
x 1: pi = 1 
x 2 : p 2 = q\ 

x 3 : p 3 =q 2 +l (2.85) 

x 4 : 0 = q3 + \qi 

* 5 : 0 =Ts + 

x 6 : 0 = f^qi + \q 3 . 


We easily find 


2 1 

Pi = q\ =q3 = o qi = -^ = 


( 2 . 86 ) 


and the approximant of order [3, 3] is 


^3,3 = 


x ~ TS* 3 

1 - F 2 ' 


(2.87) 


This expression reproduces the tangent quite well (Fig. 2.4). Its pole at \/T0/2 ■ 
1.581 is close to the pole of the tangent function at tt/2 ~ 1.571. 


2.4.2 Barycentric Rational Interpolation 


If the weights of the barycentric form of the interpolating polynomial (2.21) are taken 
as general parameters u, ^ 0 it becomes a rational function 


R(x) = 



( 2 . 88 ) 
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Fig. 2.4 (Pade 
approximation to tan(x)) 

The Pade approximant (2.87, 
dash dotted cur\<e) 
reproduces the tangent (full 
curve ) quite well 



which obviously interpolates the data points since 

lim R(x) = (2.89) 

X^Xi 


With the polynomials 1 


p{x) — y.u, fl (x - Xj) = Yujj ^ (a) 

<=0 j=0;j& i =0 

n n n , N 

Z i—r ^t oj } 

Hi || (X-Xj) = — 


a rational interpolating function is given by 2 


R(x) = 


P(x) 

fiW 


Obviously there are infinitely different rational interpolating functions which differ 
by the weights u = (uq, u\ ... u n ) (an example is shown in Fig. 2.5). To fix the 
parameters u additional conditions have to be imposed. 


2.4.2.1 Rational Interpolation of Order [M, N\ 

One possibility is to assume that P(x) and Q(x ) are of order < M and < N, 
respectively with M + N = n. This gives n additional equations for the 2 (n + 1) 


l u(x) = ECU* “ *«) as in (2-39). 

2 It can be shown that any rational interpolant can be written in this form. 
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Fig. 2.5 (Rational interpolation) The data points (1, j), (2, A), (3, ^j) are interpolated by several 

rational functions. The [1,1] approximant (2.95) corresponding to u = (5, —20, 15) is shown by 

the solid curve, the dashed curve shows the function R(x) = ,- 8 l which is obtained for 

10(3jc“ — 12*+11) 

u = (1, 1, 1) and the dash dotted curve shows the function R(x) = which follows for 

u = (1, — 1, 1) and has no real poles 


polynomial coefficients. The number of unknown equals n + 1 and the rational inter- 
polant is uniquely determined up to a common factor in numerator and denominator. 

Example Consider the data points /( 1) = f{ 2) = 3) = 

The polynomials are 


' 5 4 3 ' 


'i i i ■ 

- Ilf) — —u 1 — - Ul 

2 u 5 10 . 

x + 

— Uf) “I - —H 1 “I - - 111 

_2 u 5 10 _ 


1 1 1 

P(x) = -u 0 (x - 2)(x - 3) + -ui(x - \)(x - 3) + —M 2 (x - l)(x - 2) 

3 1 

— 3»0 + —Ml + —M 2 + 


Q{x) = uq (x — 2) (jr — 3) + ui(x — 1)(jc — 3) + u 2 (x — 1)(jc — 2) 

2 

= 6ug + 3;ri + 2«2 + [—5 mo — 4m i — 3 m2] x + [mq + u\ + U 2 \x 

To obtain a [1, 1] approximant we have to solve the equations 

1 1 1 

-mo H—in H-iii = 0 

2 5 10 “ 


MO + Ml + U2 = 0 

which gives 


x 

(2.90) 

(2.91) 

(2.92) 

(2.93) 


u 2 = 3mq «i = —4 mq 


(2.94) 
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and thus 


R(x) = 


5 Mo - 5 u 0 x 
2uqx 


6 — x 
lOx ' 


(2.95) 


General methods to obtain the coefficients m, for a given data set are described in 
[12, 13]. They also allow to determine unattainable points corresponding to m, = 0 
and to locate the poles. Without loss of generality it can be assumed [ 1 3] that M > N / 
Let P(x) be the unique polynomial which interpolates the product f(x)Q(x) 


P(xj) = f (xi)Q(xi) i = 0... M. 
Then from (2.31) we have 


(2.96) 


f (x)Q(x) - P(x) = (fQ)[x o---x M ,x](x - x 0 ) ■ ■ ■ (x -x M ). (2.97) 


Setting 

x = Xj i = M + 1,... n (2.98) 

we have 


/ (xi)Q(xi) - P(xi) = ( fQ)[x 0 . ,.x M , x,](x t - x 0 ) ...(x- x m ) (2.99) 


which is zero if P(xj)/Q(Xj) — f) for i = 0But then 

(f Q)[xo ■ ■ ■ xm, Xj] = 0 i —Mn. (2.100) 

The polynomial Q(x ) can be written in Newtonian form (2.30) 

N i -1 

Q(x) = ) = VQ + Vi (x - xo) H-1- v N (x - a'q) . .Ax - xn-\)- 

i=0 ;=0 

( 2 . 101 ) 


With the abbreviation 

gj(x) = x - Xj j — 0... N 


( 2 . 102 ) 


we find 


3 The opposite case can be treated by considering the reciprocal function values !//(*;). 
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, r n ^ f( x k)g( x k) v 

U '* k(Xk - Xr) k-0..7r.i.kfj n rfk.r^j <** - x r) 


f( x k ) 


= f[XO---Xj-l,Xj + l ...X M ,Xi\ 


(2.103) 


which we apply repeatedly to (2. 100) to get the system of n — M — N equations for 
IV + 1 unknowns 


N 

^ Ujf[Xj, Xj+ 1 ... xm, x,] = 0 i = M + 1... n (2.104) 

j =o 

from which the coefficients Vj can be found by Gaussian elimination up to a scaling 
factor. The Newtonian form of Q(x) can then be converted to the barycentric form 
as described in [6]. 


2.4.2.2 Rational Interpolation without Poles 

Polynomial interpolation of larger data sets can be ill behaved, especially for the case 
of equidistant x— values. Rational interpolation without poles can be a much better 
choice here (Fig. 2.6). 

Berrut [15] suggested to choose the following weights 
«* = (- 1 )*. 

With this choice Q(x) has no real roots. Floater and Horman [11] used the different 
choice 


Fig. 2.6 (Interpolation of a 
step function) A step 
function with uniform 
x -values ( circles ) is 
interpolated by a polynomial 
(full cuive ), a cubic spline 
(dashed cuixe) and with the 
rational Floater-Horman 
d = 1 function (2.105, 
dash-dotted curve). The 
rational function behaves 
similar to the spline function 
but provides in addition an 
analytical function with 
continuous derivatives 



x 
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Table 2.1 Floater-Horman 
weights for uniform data 


|k*I 

d 

1,1,1..., 1,1,1 

0 

1,2,2, 2, ...,2,2,2, 1 

1 

1,3, 4, 4, 4, ...,4,4,4, 3, 1 

2 

1,4,7, 8,8, 8, .... 8, 8, 8,7,4, 1 

3 

1,5, 11, 15, 16, 16, 16.16, 16, 

16, 15, 11,5, 1 

4 


«* = (-1)* -1 (---+---) k = l ... n 1 

V^+l %k %k %k —1 / 

«0 =- — U n = {- 1)"- 1 -— - (2.105) 

X\ Xo x n X n —\ 

which becomes very similar for equidistant x-values. 

Floater and Horman generalized this expression and found a class of rational 
interpolants without poles given by the weights 

min (k,n—d) i+d ^ 

.."(-l ) 1 -' 1 Z FI TT-, ( 2100 ) 

i=max(k— 4,0) j=i,j^k ^ 

where 0 < d < n and the approximation order increases with d. In the uniform case 
this simplifies to (Table 2.1) 


ma x(k,n—d) 

u k = {-\f- d jr 

i=min(k—d,0) 




(2.107) 


2.5 Multivariate Interpolation 

The simplest 2-dimensional interpolation method is bilinear interpolation . 4 It uses 
linear interpolation for both coordinates within the rectangle x,- < x < x !+ i y, < 

yi < yi+i- 


p(Xi + h x , yi + hy) = p(.Xj + h x , y t ) + h y 
f(Xi+\, y/) - f(Xi, yi) 


p(xi + h x , >' i+ i) - p(xj + h x , yt) 


yi +i - yi 


= f(Xi, yi) + h x 


X i+ 1 - Xi 


(2.108) 


4 Bilinear means linear interpolation in two dimensions. Accordingly linear interpolation in three 
dimensions is called trilinear. 
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Fig. 2.7 Bispline 
interpolation 



+/t v 


f(xi, y,:+ 1 ) + /t. 


/(.I, + i.V,'+i)-/(x,,Vi+i) 


- f(xi, y,-) - /!. 


f(x i+ i,yi)-f(Xi,yi) 

Xi+l-Xi 


y >+1 - y< 


which can be written as a two dimensional polynomial 


p(Xi + h x , y ( - + /z v ) = aoo + a l0 h x + a 0l h y + a n h x h y 


(2.109) 


with 


floo = f(xi, yi) 

f(x i+ 1, y,) - /(x ; , y ; ) 

«io = - 

Xi + i - X/ 

/(x ; , y,+i) - fixi, y t ) 

aoi = - 

yt+i - y< 

/(x,+i, y i+ i) - /(x ; , y, + i) - /(x,+i, y t ) + f(x h y,) 
a\\ - 

(X;+1 - x,)(y !+ i - y,) 


( 2 . 110 ) 


Application of higher order polynomials is straightforward. For image processing 
purposes bicubic interpolation is often used. 

If high quality is needed more sophisticated interpolation methods can be applied. 
Consider for instance two-dimensional spline interpolation on a rectangular mesh of 
data to create a new data set with finer resolution' 1 


fij = fdh x , jh y ) with 0 < i < N x 0 < j < N y . (2.111) 

First perform spline interpolation in x-direction for each data row j to calculate new 
data sets 


fi'j = s{x,, f u , 0 <i <N X ) 0 <j< N y 0 < i' < N' x (2.112) 

and then interpolate in y direction to obtain the final high resolution data (Fig. 2.7) 
f'.y = s(y y , fi'j, 0 < j < N y ) 0 <i'<N' x 0 < f < N' y . (2.113) 


5 A typical task of image processing. 
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Problems 


Problem 2.1 Polynomial Interpolation 

This computer experiment interpolates a given set of n data points by 
• a polynomial 


p(x)=^f\ n 

i= 0 k=(i,k^i 


X - x k 
Xi - Xk ' 


• a linear spline which connects successive points by straight lines 


s,(x) = a, + b, (x - xi) for x,- <x< x l+ \ 


• a cubic spline with natural boundary conditions 


(2.114) 


(2.115) 


■s(x) = Pi(x) = a, + (3i(x - Xi) + 7;(* - x t ) 2 + S,(x - x,) 3 x,- < x < x i+l 

(2.116) 

i"(x„) = s"(x 0 ) = 0 (2.117) 


• a rational function without poles 


R(x) = 



with weights according to Berrut 
«* = (-!)* 


or Floater-Hormann 

u k = (-\) k ~ l ( ---+---) k — \. .n — \ 

\-^A:+l %k %k %k —1 / 

MO =--- M„ = (- I)""'----. 

X\ Xq X n X n —\ 


(2.118) 


(2.119) 


( 2 . 120 ) 

( 2 . 121 ) 


Table 2.2 Zener diode voltage/current data 


Voltage 

-1.5 

-1.0 

-0.5 

1 0.0 

Current 

-3.375 

-1.0 

-0.125 

I 0.0 
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Table 2.3 Additional voltage/current data 


Voltage 

1.0 

2.0 

3.0 

4.0 

4.1 

4.2 

4.5 

Current 

0.0 

0.0 

0.0 

0.0 

1.0 

3.0 

10.0 


Table 2.4 Pulse and step function data 


X 

-3 

-2 

-1 

0 

1 

2 

3 

y pulse 

0 

0 

0 

1 

0 

0 

0 

ystep 

0 

0 

0 

1 

1 

1 

1 


Table 2.5 Data set for two-dimensional interpolation 


X 

0 

l 

2 

0 

1 

2 

0 

1 

2 

y 

0 

0 

0 

1 

1 

1 

2 

2 

2 

f 

i 

0 

-1 

0 

0 

0 

-1 

0 

1 


• Interpolate the data (Table 2.2) in the range 


— 1.5 < x <0. 

• Now add some more sample points (Table 2.3) for —1.5 < x < 4.5 

• Interpolate the function f(x) — sin(x) at the points x = 0, |, n, y-, It:. Take 
more sample points and check if the quality of the fit is improved. 

• Investigate the oscillatory behavior for a discontinuous pulse or step function as 
given by the data (Table 2.4) 

Problem 2.3 Two-dimensional Interpolation 

This computer experiment uses bilinear interpolation or bicubic spline interpolation 
to interpolate the data (Table 2.5) 
on a finer grid Ax = Ay = 0.1. 




































Chapter 3 

Numerical Differentiation 


For more complex problems analytical derivatives are not always available and have 
to be approximated by numerical methods. Numerical differentiation is also very 
important for the discretization of differential equations (Sect. 12.2). The simplest 
approximation uses a forward difference quotient (Fig. 3.1) and is not very accurate. A 
symmetric difference quotient improves the quality. Even higher precision is obtained 
with the extrapolation method. Approximations to higher order derivatives can be 
obtained systematically with the help of polynomial interpolation. 


3.1 One-Sided Difference Quotient 


The simplest approximation of a derivative is the ordinary difference quotient which 
can be taken forward 


d f Af f(x + h) - f(x) 

— (x) «« — =--- 

d.r Ax h 

or backward 

d/ s 4/ = fix) - ,f(x - h) 

d.r Ax h 

Its truncation error can be estimated from the Taylor series expansion 


(3.1) 


(3.2) 


fix + h) - fix) _ fix) + hf'jx) + y f"(x) H- fix) 

h li 

= f'(x) + ^f"(x) + -- - . (3.3) 


The error order is Oih). The step width should not be too small to avoid rounding 
errors. Error analysis gives 
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Fig. 3.1 (Numerical differentiation) Numerical differentiation approximates the differential 
quotient by a difference quotient ^ 37 ■ However, approximation by a simple forward difference 
37 (-Vo) ^ /U0-W-/U0) , j s not ver y accura te 


Af = fl- {fix + h){ 1 + ef), /(*)( 1 + £ 2 )) 

= (4/ + /(* + A)ei - /(x)e 2 )(l + £ 3 ) 

= 4/ + 4/S3 + /(* + h)si - f{x)s 2 H- (3.4) 


fU(Af,h{l + s 4 )) = 


4/ + 4/£ 3 + /(* + A)ei - /U)£2 

/?(! + £4) 


(1 + £5) 


_ 4/,, , , , , fix + h ) /(x) ^ 

— -7 —(1 + £5 ~ £4 + £3) H-;-£1-;—£2- ( 3 . 5 ) 

n n n 

The errors are uncorrelated and the relative error of the result can be estimated by 


4 £ _ At 

Ax Ax 

AZ 

Ay 


< 3ei 


fix) 

AZ 


£m 


(3.6) 


Numerical extinction produces large relative errors for small step width h. The opti¬ 
mal value of h gives comparable errors from rounding and truncation. It can be found 
from 

h 2 s m 

xl/"WI = |/(x)|-4. (3.7) 

2 h 

Assuming that the magnitude of the function and the derivative are comparable, we 
have the rule of thumb 


h a = sf^M ~ 10 8 


(double precision). The corresponding relative error is of the same order. 
















3.2 Central Difference Quotient 
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Fig. 3.2 (Difference 
quotient) The central 
difference quotient (Right 
side) approximates the 
derivative ( dotted ) much 
more accurately than the 
one-sided difference quotient 
(Left side) 


f(x) 



f(x) 



3.2 Central Difference Quotient 

Accuracy is much higher if a symmetric central difference quotient is used (Fig. 3.2): 

Af = fix + |) - fix - f) 

Ax h 

fix) + \f\x ) + £/"(*) + • • • - (fix) - \f'ix) + £ fix) + • • • ) 

h 

= /'W+~/"'W + -. (3.8) 

The error order is Oih 2 ). The optimal step width is estimated from 
h 1 2s m 

^- A \f"'ix)\ = \fix)\-^ (3.9) 

24 h 

again with the assumption that function and derivatives are of similar magnitude as 
h 0 = ^48% « 10" 5 . (3.10) 

h 2 _ii 

The relative error has to be expected in the order of ^ sa 10 . 


3.3 Extrapolation Methods 


The Taylor series of the symmetric difference quotient contains only even powers 
of h: 


Dih) = 


fix + h) - fjx 
2h 


h) , h 2 h 4 ,,, 

— = fix) + -/'"(*) + ~f 5 \x) + ■ ■ 


(3.11) 
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Fig. 3.3 (Numerical 
differentiation) The 
derivative sin(jc) is 
calculated numerically using 
algorithms with increasing 
error order (3.1, 3.8, 3.14, 
3.18). For very small step 
sizes the error increases as 
h ~ 1 due to rounding errors 
(Problem 3.1) 



The Extrapolation method [16] uses a series of step widths, e.g. 


h i+ 1 = 


K 

2 


(3.12) 


and calculates an estimate of D( 0) by polynomial interpolation (Fig. 3.3). Consider 
Do — D(h 0 ) and D\ — D( y). The polynomial of degree 1 (with respect to h 2 ) 
p(h ) = a + bh 2 can be found by the Lagrange method 


h 2 

p{h) = Do — 

K 


h l 

4 


K 

4 


D , 


h 2 — h q 

'A _ h 2 ' 

4 "0 


(3.13) 


Extrapolation for h = 0 gives 


p(0) = ~D 0 + ^D l . 


Taylor series expansion shows 

P( 0) = ~ (fix) + + §/ (5) c^) + ■••) + 

+ 5( /,W + i^i rw + i^5t /e>W + -) 

= fix)-~f ( 5 \x) + --- 


(3-14) 


(3.15) 

(3.16) 


that the error order is 0(Iiq). For 3 step widths ho = 2h\ = 4li2 we obtain the poly¬ 
nomial of second order (in h 2 ) 







3.3 Extrapolation Methods 
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= (* 2 - h 4)(h 2 ~ ’j) (/* 2 -/? 2 )(/t 2 -|) (h 2 -h 2 )(h 2 

°(h 2 - %(h 2 - ’ ,2 °) 1 (^-fc 2 )(*Q-*!) Vo _/ 1 2 )( , 'o 

l"o 4 P"o W 4 VW 16 1 (.15 "0' )( -T5 


A o, 

4 J 


and the improved expression 


P( 0) = D 0 


64 


3 15 

4 ’ 16 


+ Dy 


16 


z 3 _3_ 
4 ’ 16 


+ D 2 


-15 ^3 

16 ’ 16 


1 4 64 * 

= tt D ° ~ g Dl + y 5 ° 2 = / <*) + 0 V)- 

Often used is the following series of step widths: 


, /in 

/17 = 4. 


'4 

4 ; 

(3-17) 


(3.18) 


(3.19) 


The Neville method 

(h 1 - h bP i+h .. k (h 2 ) - (4 - g)P,(A 2 ) 


gives for h=0 

Pi-k-l 


(3.20) 


"0 20 

2 k 2 ' 


/V-4- = 


■ i+l-ifc 


1 - 2 *-' 

which can be written as 

p _ p I Pi-'k—l Pi+\—k 

Pi... k - P i+h .-k + -! _ 2 *_i- 

and can be calculated according to the following scheme: 

P) = D(h 2 ) Pot P 012 P 0123 
Pi = £>(y) P 12 P 123 
Pi = D& 43 


(3.21) 


(3.22) 


(3.23) 


Here the values of the polynomials are arranged in matrix form 


Pi k = Ti k-i = Ti 


(3.24) 
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with the recursion formula 


Tij — + 


Tij -1 


Ti 


+1,7 


1 -2 J 


(3.25) 


3.4 Higher Derivatives 

Difference quotients for higher derivatives can be obtained systematically using 
polynomial interpolation. Consider equidistant points 


x n = xq + nh = ■ ■ ■ xo — 2h, xq — h, xo, xo + h , xo + 2h , • • • . (3.26) 


From the second order polynomial 


(x-x 0 )(x-xi) (x - x_i)(x - Xi) 

p(x) = y_i --—-- + yo- 


+ yi 


(X— 1 - x 0 )(x_i - Xi) 
(x - x_i)(x - Xo) 

(xi - x_i)(xi - xo) 


(x 0 - x_i)(x 0 - Xl) 


(x — Xo)(x — xi) (x — x_i)(x — xi) 

y-i-^-+ yo - 


+ yi 


2 h 2 

(x - x_i)(x - x 0 ) 


—h 2 


2 h 2 

we calculate the derivatives 


2x — xo — xi 2x — x_i — xi 2x — x_i — xo 

p w = y-i —— + yo-r?— + yi- 


2 h 2 

, y-1 O yo.yt 

which are evaluated at xo: 


- h 2 


2 h 2 


w. . . 1 , 1 f(x o + h ) - /(x 0 - h) 

/ (.vo) * p (VO) = - Yh y-, + s ,., =- Yh - 


/"(xo) « p"(xo) = 


/(x 0 - h) - 2/(x 0 ) + /(x 0 + h ) 
h 2 


(3.27) 

(3.28) 

(3.29) 

(3.30) 

(3.31) 


Higher order polynomials can be evaluated with an algebra program. For five sample 
points 


xo — 2 h, xo — h, xo, xo + h, xq + 2 h 
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we find 


/'(*o) « 
/"(*o) » 


f(x o - 2h) - 8/(xp - h) + 8/Ocq + /r) - /(xq + 2/Q 
12 h 

-fix o - 2ft) + 16/(xq - h) - 30/ (xq) + 16/(xp + /Q 

12/r 2 


(3.32) 
fix 0 + 2h) 

(3.33) 


/"Vo) ; 

/ (4) (x 0 ) 


-fix o - 2 h) + 2/(xp - ft) - 2/(xp + ft) + /(xp + 2ft) 

2 ft 3 ' 

/(xp - 2ft) - 4/(xp - ft) + 6/(x 0 + ft) - 4/(x 0 + ft) + /(xp + 2ft) 

ft 4 


3.5 Partial Derivatives of Multivariate Functions 


Consider polynomials of more than one variable. In two dimensions we use the 
Lagrange polynomials 


Li,j(x,y) = Y[ 

k^i 


(x — Xk) T—r 

(Xi - X k ) xx 


0 - yi) 
iyj - yiY 


The interpolating polynomial is 


p(x, y) = ^fijLijix, y). 
ij 


For the nine sample points 

ix-i,y\) (x 0 , yi) ixi,y\) 
ix~ i,y 0 ) (x 0 ,yo) (-ri,.vo) 

(x-i, y_i) (xo, y_i) (xi,y_i) 
we obtain the polynomial 


P(x, y) = f- i,-i 


jx - x 0 )(x - Xi )(y - yo)iy - yi) 

(x~\ - x 0 )ix -1 - xi)(y_i - y 0 )(y_i) - y\) 


+ ••• 


which gives an approximation to the gradient 


(3.35) 


(3.36) 


(3.37) 


(3.38) 


grad/(x 0 yo) 


gradp(x 0 y 0 ) = 


( 


f(xo+h,yo)-f(x 0 -h,yo) 

2h 

f(xo,yo+h)-f(xo,yo-h ) 
2/7 


)■ 


(3.39) 
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the Laplace operator 


/ a 2 a 2 \ (d 2 d 2 \ 

[d^ + d?) f(x °’ yo) + a?) p(x °' 

= ^2 ( /(*O' yo + h) + fix 0, yo - h) + fix 0, y 0 


yo) 

+ h) + /(xo, Vo - A) - 4/Uo, yo)) 

(3.40) 


and the mixed second derivative 


3 2 

3x3y 


/(.vo, vo) 


a 2 

3x3 v 


p(vo, yo) 


1 

= (/(vo + h, yo + h) + /(xo - /!, vo - h) - /(x 0 - /i, yo + h) - /(x 0 + A, vo - *)) • 

(3.41) 


Problems 


Problem 3.1 Numerical Differentiation 

In this computer experiment we calculate the derivative of fix) = sin(x) numerically 
with 

• the single sided difference quotient 


d f _ fix + h) - fix) 
dv h 

• the symmetrical difference quotient 


(3.42) 


d/ _ „ ,, , fix + h) - fix - h) 
d7 ” Dhf(x) = - 2h -’ 


(3.43) 


• higher order approximations which can be derived using the extrapolation method 


- l -D h fix) + ^D h/2 fix) (3.44) 

^Dhfix) - ^ D h/2 fix) + ( ^D h/4 fix). (3.45) 


The error of the numerical approximation is shown on a log-log plot as a function of 
the step width h. 






Chapter 4 

Numerical Integration 


Physical simulations often involve the calculation of definite integrals over 
complicated functions, for instance the Coulomb interaction between two electrons. 
Integration is also the elementary step in solving equations of motion. 

An integral over a finite interval [a, b] can always be transformed into an integral 
over [0. 1] or [—1, 1] 


f(x)dx = / f(a + (b — a)t ) (b — a)dt 

Jo 

' a + b b — a \ b — a 


" 1 / a + b b — a \ 

J \~ + ~ t ) 


-dt. 


(4.1) 


An Integral over an infinite interval may have to be transformed into an integral 
over a finite interval by substitution of the integration variable, for example 


f(x)dx = 

i 

f(x)dx 


7 Vi -t) (i - 1) 2 
J-ZKi-t 2 ) (t 2 — l; 


-dt. 


(4.2) 

(4.3) 


In general a definite integral can be approximated numerically as the weighted 
average over a finite number of function values 


f(x)dx ~ y. Wifi*). 


(4.4) 


Specific sets of quadrature points x, and quadrature weights w / are known as “inte¬ 
gral rules”. Newton-Cotes rules like the trapezoidal rule, the midpoint rule or 
Simpson’s rule, use equidistant points x, and are easy to apply. Accuracy can 
be improved by dividing the integration range into sub-intervals and applying 
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4 Numerical Integration 


composite Newton-Cotes rules. Extrapolation methods reduce the error almost to 
machine precision but need many function evaluations. Equidistant sample points 
are convenient but not the best choice. Clenshaw—Curtis expressions use non uniform 
sample points and a rapidly converging Chebyshev expansion. Gaussian integration 
fully optimizes the sample points with the help of orthogonal polynomials. 


4.1 Equidistant Sample Points 


For equidistant points 


Xi = a + ih 


i = 0 ... N h = 


b — a 
N 


(4.5) 


the interpolating polynomial of order N with p (x t ) = fix,) is given by the Lagrange 
method 


p(x) = n 


■ *k 


i —0 k=0,k^i 


Xi ~ X k 


(4.6) 


Integration of the polynomial gives 


r b N f h N r - r, 

/ p(x)dx=Y J /i n f~v dx - 

Ja i= 0 Ja k=0.k& ' k 

After substituting 

x = a + hs 
x — Xk = h (s — k) 

Xj — Xk = (i — k)h 
we have 



and hence 


, N 

p(x)dx = (b- a) y. fm. 


(4.7) 


(4.8) 


(4.9) 


(4.10) 







4.1 Equidistant Sample Points 
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The weight factors are given by 

u), = (b — a)oij = Nhctj. (4.11) 

4.1.1 Closed Newton-Cotes Formulae 

For N = 1 the polynomial is 


X — Xi X — Xa 

p(x) = fo - L + /i-- 

Xo — X'l XI — Xo 

and the integral is 

- 1 S -1 


(4.12) 


p(x)dx = fa 


= ~foh\ 


0-1 
(1 — l ) 2 ( 0 - 1) 2 


hds + fi 


" S -^hds 

) i-o 

2 




= h 


fo + fi 


(4-13) 


which is known as the trapezoidal rule (Fig. 4.1). N — 2 gives Simpson’s rule 

2h f b + 4 / 1 + / 2 > (4 . 14) 


Larger N give further integration rules 

fo + 3/t + 3 /t + fj 


3/7 


8 


4/i 


7/q + 32/ t + 12/ 2 + 32/ 3 + 7/4 
90 


3 /8 — rule 


Milne — rule 


5 h 


19/q + 75/ t + 50/2 + 50/3 + 75/4 + 19/5 
288 


41/o + 216/! + 27/ 2 + 272/s + 27 / 4 + 216/ s + 41/ 6 
6 h -—r- Weddle — rule. 


840 


(4-15) 


For even larger N negative weight factors appear and the formulas are not 
numerically stable. 
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f(a) f(b) 



Fig. 4.1 (Trapezoidal rule and midpoint rule) The trapezoidal rule {Left) approximates the integral 
by the average of the function values at the boundaries. The midpoint rule (Right) evaluates the 
function in the center of the interval and has the same error order 


4.1.2 Open Newton-Cotes Formulae 


Alternatively, the integral can be computed from only interior points 

b — a 

Xj = a + ih i — 1,2,... N h = 


N+ 1 


The simplest case is the midpoint rule (Fig. 4.1) 

J f (x)dx 2 hfi = (b - 

The next two are 

y(/i + / 2 ) 

y ( 2/ 1 -/ 2 + 2 / 3 J . 


(4.16) 


(4.17) 


(4.18) 


(4.19) 


4.1.3 Composite Newton-Cotes Rules 

Newton-Cotes formulas are only accurate, if the step width is small. Usually the 
integration range is divided into small sub-intervals 


[xj, x, + i] x t — a + ih i = 0 ... N 


(4.20) 










4.1 Equidistant Sample Points 
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for which a simple quadrature formula can be used. Application of the trapezoidal 
rule for each interval 

= T (/(*■■) + /(* + l)) (4-21) 

gives the composite trapezoidal rule 

T = /! (+ fia + h) + • • • f(b - h ) + (4.22) 

with error order 0(h 2 ). Repeated application of Simpson’s rule for [a, a + 2h], [a + 
2h, a + 4h] ... gives the composite Simpson’s rule 

5 = h - (f(a) + 4 /(a + h ) + 2 f(a + 2h) + 4 f(a + 3 h)+ 

■■■ + 2f(b -2h) + 4f{b - h) + f(b)\ (4.23) 

with error order 0(h 4 ). ] 

Repeated application of the midpoint rule gives the composite midpoint rule 
M = 2h^f{a + h) + f(a + 3 h) + • ■ ■ f(b — /i)^ (4.24) 

with error order 0(h 2 ). 


4.1.4 Extrapolation Method (Romberg Integration) 


For the trapezoidal rule the Euler-McLaurin expansion exists which for a 2 in times 
differentiable function has the form 


f{x)dx - T = a 2 h A + + ■■■ a 2m - 2 h 2m ~ 2 + 0(h 2m ) 


l 2m \ 


(4.25) 


Therefore extrapolation methods are applicable. From the composite trapezoidal rule 
for h and h/2 an approximation of error order 0{h 4 ) results: 


1 The number of sample points must be even. 
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/ f(x)dx — T (h) = aih 2 + 0 : 4 / 1 4 + • • ■ 
h 2 h 4 

f(x)dx - T(h/2) = a 2 — +«4 — H- 


w 4r(A/2) - r(/t) 
/ ( x)dx --- 


- 0 : 4 - 


(4.26) 

(4.27) 

(4.28) 


More generally, for the series of step widths 


, ho 
k ~ 2 k 


(4.29) 


the Neville method gives the recursion for the interpolating polynomial 


„ n 2\ ( h ~ ^)Pi + x -k(h 2 ) - (h 2 - ^)Pi... k -,(h 2 ) 

P ‘ -k(h ) = ---p-*2 - 

"0 _ 2o_ 

2 lk 2 2i 


(4.30) 


which for h = 0 becomes the higher order approximation to the integral (Fig. 4.2) 


Pi k = 


2~ 2k Pi - 2~ 2i Pj + \... k Pi... k _1 - 2 Zk ~ 2 'P i+1 ... k 


— Pi+l-k + 


2 ~ 2k - 2~ 2 ' 

Pi"-k— 1 Pi+\---k 

1 - 2 2k ~ 2i ' 


1 _ 2 2k ~ 2i 


(4.31) 



Fig. 4.2 (Romberg integration) The integral sin(.r 2 )d.t: is calculated numerically. Circles show 
the absolute error of the composite trapezoidal rule (4.22) for the step size sequence /i;+i = hi /2. 
Diamonds show the absolute error of the extrapolated value (4.31). The error order of the trapezoidal 
rule is 0(h 2 ) whereas the error order of the Romberg method increases by factors of h 2 . For very 
small step sizes the rounding errors dominate which increase as h~ l 
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The polynomial values can again be arranged in matrix form 

Pq Poi P012 ■ ■ ■ 

Pi Pi 2 

P 2 (4.32) 


with 


T U ~ P i -i+j 

and the recursion formula 



Tij = Ti+i,j-i + 


Tu-i-Ti 


+1,7-1 


1 - 2 2 i 


(4.33) 

(4.34) 

(4.35) 


4.2 Optimized Sample Points 

The Newton-Cotes method integrates polynomials of order up to N — 1 exactly, 
using N equidistant sample points. Unfortunately the polynomial approximation 
converges slowly, at least for not so well behaved integrands. The accuracy of the 
integration can be improved by optimizing the sample point positions. Gaussian 
quadrature determines the N positions and N weights such, that a polynomial of order 
2 N — 1 is integrated exactly. The Clenshaw-Curtis and the related Fejer methods 
use the roots or the extrema of the Chebyshev polynomials as nodes and determine 
the weights to integrate polynomials of order N. However, since the approximation 
by Chebyshev polynomials usually converges very fast, the accuracy is in many 
cases comparable to the Gaussian method [17, 18]. In the following we restrict the 
integration interval to [—1, 1]. The general case [«, b] is then given by a simple 
change of variables. 


4.2.1 Clenshaw-Curtis Expressions 

Clenshaw and Curtis [19] make the variable substitution 


x = cos 6 dx = — sin 0 d6 


(4.36) 
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for the integral 
/> 1 


f{x)clx — / /(cos r) sin t dt 


(4.37) 


/-i 


and approximate the function by the trigonometric polynomial (7.19 with 
N = 2 M, T = 2tc) 


1 1 VT 1 

/(cos 0 = ^c 0 + — 2 Cj cos (j t ) + —c M cos (Mt) 

7=1 

which interpolates (Sect. 7.2.1 ) /(cos 7) at the sample points 
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t n — nAt — 

n — 

with/; = 0, 1,.. 

.. M 

M 





/ 7T \ 


x n = cos t n 

= cos 

In — ) 



V M) 



and where the Fourier coefficients are given by (7.17 ) 

M -i 

Cj = /o + 2X /(cos(f„)) cos(-yn) + f M cos(yjr). 

n=l 


(4.38) 


(4.39) 

(4.40) 


(4.41) 


The function cos(j f ) is related to the Chebyshev polynomials of the first kind 
which for — 1 < x < 1 are given by the trigonometric definition 


Tj(x ) = cos (j arccos(x)) 

(4.42) 

and can be calculated recursively 

T 0 (x) = 1 

(4.43) 

/ (x) = X 

(4.44) 

Tj+iix) = 2x Tj(x) - Tj_i(x). 

(4.45) 

Substituting x = cos t we find 

7) (cost) = cos (j t). 

(4.46) 


Hence the Fourier series (4.38) corresponds to a Chebyshev approximation 
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fix) = 7)(*) = ^-T 0 ix) + X %Tj{x ) + ^.T m (x) 


Cm 


2 M 

j= 0 7=1 

and can be used to approximate the integral 


M 


2 M 


(4.47) 



f(x)dx 



1 1 ^ 

^o + -2 j c j co S (;/) + 


M 


7 = 1 


1 

-cm cos(Mf) 


sin 6 dO 
(4.48) 



cos(jjz) + 1 

1 -j 2 


1 cos(Mjt) + 1 
2M Cm 1 - M 2 


(4.49) 


where, in fact, only the even j contribute. 

Example Clenshaw Curtis quadrature for M — 5 

The function has to be evaluated at the sample points Xk = cos( jk) = (1, 0.80902, 
0.30902, —0.30902, —0.80902, —1). The Fourier coefficients are given by 


/c°\ 


111 2 2 2 1 \ 


fo\ 

Cl 


1 1.618 0.618 -0.618-1.618-1 


h 

Cl 


1 0.618 -1.618 -1.618 0.618 1 


h 

C3 


1-0.618-1.618 1.618 0.618 -1 


h 

C4 


1 -1.618 0.618 0.618 -1.618 1 


h 

\c 5 ) 


\1 -2 2 -2 2 -l) 


\/s/ 


and the integral is approximately 


jj{x)dx*{ i 0 —^ 0—^0) 


Cl 

Cl 

C3 

C4 


(4.50) 


= 0.0400/o + 0.3607/i + 0.5993 f 2 + 0.5993 / 3 + 0.3607 f 4 + 0.0400/ 5 . 

(4.51) 

Clenshaw Curtis weights of very high order can be calculated efficiently [20, 21] 
using the FFT algorithm (fast Fourier transformation, Sect. 7.3.2). 
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4.2.2 Gaussian Integration 


Now we will optimize the positions of the N quadrature points x, to obtain the 
maximum possible accuracy. We approximate the integral by a sum 


f (x)dx 


N 

y' f(x i )w l 

i =1 


(4.52) 


and determine the 2 N parameters X; and vu, such that a polynomial of order 2 N — 1 
is integrated exactly. This can be achieved with the help of a set of polynomials which 
are orthogonal with respect to the scalar product 

< fg >= [ f (x)g(x)w(x)dx (4.53) 

J a 

where the weight function w(x) and the interval [a, b] determine a particular set of 
orthogonal polynomials. 


4.2.2.1 Gauss-Legendre Integration 

Again we restrict the integration interval to [—1, 1] in the following. For integrals 
with one or two infinite boundaries see Sect. 4.2.2. The simplest choice for the weight 
function is 


w(x)= 1. (4.54) 

An orthogonal system of polynomials on the interval [—1, 1] can be found using the 
Gram-Schmidt method: 

Po = 1 (4.55) 


P\ = x — 


Po 


< PqPq > 


x P(){x)dx = x 


Pi = x- 


P l 

< Pi Pi > 


x"P\ (x)dx 


Po 

< PqPq > 


x~Po(x)dx 


x — 


1 

3 


(4.56) 


(4.57) 





4.2 Optimized Sample Points 


57 


P n = x" 


Pn -, 


< Pn—] Pn — ] > 


x n P„-i(x)dx 


n— 1mi~ 1 X' 7—1 


n—2 


< P n - 2 P„-2 > J- 


x" P n _ 2 (x)dx 


(4.58) 


The P„ are known as Legendre-polynomials. Consider now a polynomial /i(x) of 
order 2N — 1. It can be interpolated at the N quadrature points x, using the Lagrange 
method by a polynomial p(x) of order /V — 1: 


P(x) = YLj(x)p(Xj). 

7=1 

Then p(x) can be written as 

p(x) = p(x) + (x - xi)(x - xt) ... (x - x N )q(x). 


(4.59) 


(4.60) 


Obviously q (x) is a polynomial of order (2 N — 1) — N = N — 1. Now choose the 
positions x, as the roots of the Legendre polynomial of order N 


( x — xi)(x — x 2 ) ■ ■ ■ (x — Xn) — Pn(x). 


(4.61) 


Then we have 

f. l 


(x — x0(x — x 2 ) ... (x — XN)q(x)dx — 0 


i-i 


since Pn is orthogonal to the polynomial of lower order. But now 




„1 N 


7=1 


7=1 


with the weight factors 


Wj = J Lj(x)dx. 


(4.62) 


p(x)dx = / p{x)dx = / Yp(xj)Lj(x)dx = Ywj P (Xj) (4.63) 


(4.64) 


Example (Gauss-Legendre integration with two quadrature points) The 2nd order 
Legendre polynomial 


P 2 (x) = x 2 


(4.65) 


has two roots 
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The Lagrange polynomials are 



and the weights 



This gives the integral rule 


/, mdx “ /(“ Jl) + /(VT)- 

For a general integration interval we substitute 
a + b b — a 


and find the approximation 



The next higher order Gaussian rule is given by 


(4.66) 


(4.67) 


(4.68) 


(4.69) 


(4.70) 


(4.71) 


(4.72) 


n = 3 : wi = W 3 = 5 / 9 , wi = s / 9 , x 3 = —x\ = 0.77459..., xi = 0. 


(4.73) 
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4.2.2.2 Other Types of Gaussian Integration 

Further integral rules can be obtained by using other sets of orthogonal polynomials, 
for instance 

Cliebyshev Polynomials 


w{x) = 


Vl - X 2 


J f(x)dx = J f(x)\/\ — x 2 w(x)dx 


T n+l (x) = 2x T n (x) - T„_ i(x) 



Hermite Polynomials 

w(x) = e~ x 


f(x)dx = / f{x)e x uu(x)dx 


(4.74) 

(4.75) 

(4.76) 

(4.77) 

(4.78) 

(4.79) 


H 0 (x) =1, H\(x) = 2x, H n+ i(x) = 2xH n (x) — 2nH n _\(x). 


Laguerre Polynomials 

w(x) = e~ x (4.80) 



f(x)dx = 



f (x)e x w(x)dx 


(4.81) 


L 0 (x) = l, L\(x) = \ — x, L n+ i(jc) 


—— ((2 n + 1 - x)L n (x) - nL n —i(x)) . 
n + 1 

(4.82) 







60 


4 Numerical Integration 


4.2.2.3 Connection with an Eigenvalue Problem 

The determination of quadrature points and weights can be formulated as an eigen¬ 
value problem [22, 23], Any set of orthogonal polynomials P n (x) with 


/ P m (x) P„(x)w(x)dx = 

J a 

satisfies a three term recurrence relation 

P„+i(x) = (a„+ix + b n+ 1 ) P n (x) - c n+ \P n -\{x) 
with a„ > 0, c„ > 0 which can be written in matrix form [24] 


(4.83) 


(4.84) 


/ Po(x) \ 
Pl(.x) 


\P N - iW/ 


j_bi _L 
“ 1 "1 
£2 _bj J_ 
a 2 a 2 a 2 


or shorter 


C N -1 b N -1 1 

“JV-l “JV-l a N -1 

c ,V h N 

a l V 


/ Po(^) \ 


Vpjv-iW/ 


/ 0 \ 
0 




(4.85) 


JtPW = TP(jf) H-Piv(^)ev-i 

a N 


(4.86) 


with a tridiagonal matrix 7\ Obviously Pn(x) — 0 if and only if 


Xj P (xj ) = 7’P(x ; ), (4.87) 

hence the roots of P V (x) are given by the eigenvalues of T. The matrix T is symmetric 
if the polynomials are orthonormal, otherwise it can be transformed into a symmet¬ 
ric tridiagonal matrix by an orthogonal transformation [24] . Finally the quadrature 
weight corresponding to the eigenvalue xj can be calculated from the first component 
of the corresponding eigenvector u ; [22] as 

t ( h 

Wj = My j x / w{x)dx. (4.88) 
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Problems 


Problem 4.1 Romberg Integration 

Use the trapezoidal rule 


z G- 


T{h) = h[ -/(a) + f (a + h) + ■ 


■ f(b-h)+-f(b)] = 


H 


f(x)dx + ■ 


(4.89) 


with the step sequence 



and calculate the elements of the triangular matrix 


(4.90) 


T(i, 0) = T(hj) 


(4.91) 


T(i, k) = T(i + l,k- 1)+ 


T{i,k- 1 )-T(i + l,k-l) 


(4.92) 


to obtain the approximations 

?01 = Pot, 7o2 = Poi2> Tq3 = Pq\23, . ■ ■ (4.93) 

• calculate 


sin (xr)dx 


0.6773089370468890331... 


(4.94) 


and compare the absolute error of the trapezoidal sums T (/;,-) = 7} o and the extrap¬ 
olated values To,;- 
• calculate 



for e = 10 3 . Compare with the composite midpoint rule 

=*(4 4) + 4+f) + ■ ■ ■+4 - f)+4 4)) 

(4.96) 



Chapter 5 

Systems of Inhomogeneous Linear Equations 


Many problems in physics and especially computational physics involve systems of 
linear equations 


a\\X\ H-h a\ n x„ = b\ 

: : : (5.1) 

ttniXi a a ii Xu — b n 

or shortly in matrix form 

Ax = b (5.2) 

which arise e.g. from linearization of a general nonlinear problem like (Sect. 22.2) 


( Fi(x\ . ..x„) 
F n (x i.. ,x n ) 


Fi(x{ 0) ...4 0) )\ /Hr • • • Hr\ 


dFi 


F n (xf ] . ..Xn’)) 


(Ok 


dF„ 
- dx\ 


x\ 


-x™\ 


dF„ 

~Fxf, / 


(0) 

yX n - x n ' 


+ ... 


(5.3) 


or from discretization of differential equations like 


„ df 
“ ()x 


( 


• g(x) 


\ 


nU+VAx)-fUAx) _ g ( jAx ) 


V 


/ 




1 J_ 

Ax Ax ■ 

_ Zx 




V 


/ 

\ 


/ 

\ 

fi 

fi+1 

- 

9j 

9j +1 

\ 






(5.4) 
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5 Systems of Inhomogeneous Linear Equations 


If the matrix A is non singular and has full rank, (5.2) can be formally solved by 
matrix inversion 

x = A _1 b. (5.5) 

If the matrix is singular or the number of equations smaller than the number of vari¬ 
ables, a manifold of solutions exists which can be found efficiently by singular value 
decomposition (Sect. 11.2). The general solution is given by a particular solution 
and the nullspace of A 

x = x p + z with Ax p = b and Az = 0. (5.6) 

If the number of equations is larger than the number of variables there exists no 
unique solution. The “bestpossible solution” can be determined by minimizing the 
residual 

| Ax — b| = min (5.7) 

which leads to a least squares problem (Sect. 11.1.1). 

In the following we discuss several methods to solve non singular systems. If 
the dimension is not too large, direct methods like Gaussian elimination or QR 
decomposition are sufficient. Systems with a tridiagonal matrix are important for 
cubic spline interpolation and numerical second derivatives. They can be solved 
very efficiently with a specialized Gaussian elimination method. Practical applica¬ 
tions often involve very large dimensions and require iterative methods. Station¬ 
ary methods apply a simple iteration scheme repeatedly. The slow convergence of 
the methods by Jacobi and Gauss-Seidel can be improved with relaxation or over¬ 
relaxation, Non-stationary methods construct a sequence of improved approxima¬ 
tions within a series of increasing subspaces of Modern Kiylov-space meth¬ 
ods minimize the residual r = Ax — b within the sequence of Krylov-spaces 
K n (A, r (0) ) = span(r (0 \ Ar (0) , ... A" _1 r (0) ). We discuss the conjugate gradients 
method (CG [25]) for symmetric positive definite matrices and the method of gen¬ 
eral minimal residuals (GMRES [26]) for non symmetric matrices. Other popular 
methods are the methods of bi-conjugate gradients ( BiCG [27] BiCGSTAB [28]), 
conjugate residuals (CR [29]) , minimal residual (MINRES [30]), quasi-minimal 
residual (QMR [31]), the symmetric LQ-method (SYMMLQ [32]) and Lanzcos type 
product methods (LTPM [33-35]). 


5.1 Gaussian Elimination Method 


A series of linear combinations of the equations transforms the matrix A into an 
upper triangular matrix. Start with subtracting an/an times the first row from rows 
2 ■■■ n 
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/af\ 


/ a " \ 

a 2 


a T 2 - fciaf 



hn-H 

C 

1 

^ SC 


which can be written as a multiplication 
A (1) = L\A 


(5.8) 


(5.9) 


with the Frobenius matrix 


L ■ = 


/ 1 
—hi 1 
—hi 1 


\-lnl 1/ 

The result has the form 


, &i 1 

k\ = —• 

a ii 


A™ = 


/All 

a 12 • 

@ln —1 \ 

0 

« (1) • 
a 22 

(1) (1) 

■ «2«-l «2„ 

0 

a (l) • 
a 32 

• 0 (1) 
a 3n 

0 



V o 

a (l) • 

a n2 

• ••• a (1) y 

“/IM / 


(5.10) 


(5.11) 


Now subtract — times the second row from rows 3 • • ■ n. This can be formulated as 

«22 


a (2) = l 2 a {1) = l 2 l { a 

with 

Z 1 \ 

0 1 

L 2 = 0 -/ 32 1 

v 0 -Ini 1/ 


In = 


2 U) 

-*/2 

I ay 

■*22 


(5.12) 


(5.13) 


The result is 
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(2) 

«12 

(2) 

«13 ‘ 

■ fl (2) \ 
"In 

0 

(2) 

«22 

(2) 

«23 • 

(2) 

• a 2n 

0 

0 

(2) 

fl 33 • 

(2) 

• «3« 

V o 

0 

(2) 

«„3 • 

' j 


Continue until an upper triangular matrix results after n— 1 steps: 

A (,!_1) = L„_ 1 A (,, - 2) 

/I 


\ 1 1 


l n.n — 1 — 


a (n ~ 2) 

a n,n -1 

Jn—Z) 


u = 


I Mil M 12 U 13 • • • U\ n \ 
M 2 2 M 2 3 • • • H 2 „ 
«33 • ‘ • M3„ 


\ J 

The transformed system of equations 


Ux = y y = • L 2 Lib 


can be solved easily by backward substitution: 
1 

x n = — y„ 

Mnn 

yn —1 X n U n —\ n 

Xn —1 = 

Un—l,n— 1 


(5.14) 


(5.15) 


(5.16) 


(5.17) 


(5.18) 


(5.19) 

(5.20) 

(5.21) 

(5.22) 


Alternatively the matrices L, can be inverted: 
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Lr 1 = 

/i 

hi i 
*31 1 

\ 

■■•^“-1 = 

/I 

1 

\ 



V 



1 

ln,n— 1 1 / 


This gives 

A = L- x l L?...L- n \U. 


(5.23) 


(5.24) 


The product of the inverted matrices is a lower triangular matrix: 

\ 


L~ l L~ l — 
u l ^2 — 


/i 

hi 1 
*31 hi 1 


\ h 1 hi 1 / 


I — L~ l / _1 . .. / -1 — 

u ^2 L ‘n -1 


/ 1 

*21 1 


o; 1.1 h — l,l ' 1 

\ In 1 *«2 ’ ’ ' h,n —1 1 ) 


(5.25) 


Hence the matrix A becomes decomposed into a product of a lower and an upper 
triangular matrix 


A = LU 


(5.26) 


which can be used to solve the system of (5.2). 


Ax = Tf/x = b 


(5.27) 


in two steps: 

Ly = b 

(5.28) 

which can be solved from the top 

yi = bi 

(5.29) 

yi — b 2 — hiyi 

(5.30) 


(5.31) 
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and 


Ux = y 

which can be solved from the bottom 
1 

x n = — y n 

Mnn 

yn —1 X n U n —\ n 
Xn —1 = • 

Mu—l,n —1 


(5.32) 


(5.33) 

(5.34) 

(5.35) 


5.1.1 Pivoting 


To improve numerical stability and to avoid division by zero pivoting is used. Most 
common is partial pivoting. In every step the order of the equations is changed 
in order to maximize the pivoting element dk,k in the denominator. This gives LU 
decomposition of the matrix PA where P is a permutation matrix. P is not needed 
explicitly. Instead an index vector is used which stores the new order of the equations 


/ 1 \ 

\ N I V'V/ 


(5.36) 


Total pivoting exchanges rows and columns of A. This can be time consuming for 
larger matrices. 

If the elements of the matrix are of different orders of magnitude it can be necessary 
to balance the matrix, for instance by normalizing all rows of A. This can be also 
achieved by selecting the maximum of 


Oik 

Z j lay I 


(5.37) 


as the pivoting element. 


5.1.2 Direct LU Decomposition 

LU decomposition can be also performed in a different order [36]. For symmetric 
positive definite matrices there exists the simpler and more efficient Cholesky method 
decomposes the matrix into the product LL T of a lower triangular matrix and its 
transpose [37], 
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5.2 QR Decomposition 

The Gaussian elimination method can become numerically unstable [38]. An alter¬ 
native method to solve a system of linear equations uses the decomposition [39] 

A= QR (5.38) 

with a unitary matrix Q} Q = 1 (an orthogonal matrix Q r Q = 1 if A is real) and 
an upper right triangular matrix R. The system of linear equations (5.2) is simplified 
by multiplication with Q ' = Q~ 1 


QRx = Ax — b 

(5.39) 

Rx = Qb. 

(5.40) 


Such a system with upper triangular matrix is easily solved (see 5.32). 


5.2.1 QR Decomposition by Orthogonalization 


Gram-Schmidt orthogonalization [2, 39] provides a simple 
decomposition. It is used for symbolic calculations and also 
(1 1.1.2) but can become numerically unstable. 

From the decomposition A = QR we have 

way to perform a QR 
for least square fitting 

k 

a ik = X Wj* 

1=1 

(5.41) 

k 

w = X ow 

i=i 

(5.42) 

which gives the A'-th column vector a* of A as a linear combination of the orthonormal 
vectors qi • • • q^. Similarly q* is a linear combination of the first k columns of A. 
With the help of the Gram-Schmidt method and q ; are calculated as follows: 

Hi := |flil 

ai 

qi := — 

r ii 

(5.43) 

(5.44) 


For k = 2, ■■■ n: 
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r,k ■= q,at i = 1 • • • k - 1 
bit := a* — ru-qi — • • • n_uq/:-i 
nk ■= |b*| 



Obviously now 


a* = Lt/tqA + /■*-!,jtq*-i H-t'uqt 


since per definition 
q,a* = r it t = 1 • • ■ A: 
and 

r lk = Ibf'l 2 = |a *| 2 + + • • ■ rj_ l k -2r\ k -2r|_ u . 

Hence 


(5.45) 

(5.46) 

(5.47) 

(5.48) 


(5.49) 


(5.50) 


(5.51) 


q^ai = —(a* — r i^qi • • • rk-i^k-x)^ = —(I«a | 2 — r\ k • ■ ■ — r k_\ a) = r kk- 

fkk r kk 

(5.52) 


Orthogonality gives 


q,aA =0 i = k + 1 • • • n. 


(5.53) 


In matrix notation we have finally 


A = (aj ■ ■ -a„) = (qi ■ ■ -q„) 


( r n n 2 ■■■ r\ n \ 
r 2 2 ■ ■ ■ r 2n 


(5.54) 


If the columns of A are almost linearly dependent, numerical stability can be 
improved by an additional orthogonalization step 


bA bA - (qibA-)qi-(qt-tb*)q*_i 


(5.55) 


after (5.46) which can be iterated several times to improve the results [2, 40]. 
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Fig. 5.1 (Householder 
transformation) 
Geometrically the 
Householder transformation 
(5.56) is a mirror operation 
with respect to a plane with 
normal vector u 


U 



5.2.2 QR Decomposition by Householder Reflections 

Numerically stable algorithms use a series of transformations with unitary matrices, 
mostly Householder reflections (Fig. 5.1) [2] 1 which have the form 

P = P r = 1 - 2uu r (5.56) 

with a unit vector 

|u| = 1. (5.57) 

Obviously P is an orthogonal matrix since 


P T P = (1 - 2uu 7 ')(1 - 2uu 7 ’) = 1 - 4uu r + 4uu^uu^ = 1. (5.58) 


In the first step we try to find a vector u such that the first column vector of A 

'flu \ 


ai = 


(5.59) 


, fln i J 


'Alternatively Givens rotations [39] can be employed which need slightly more floating point 
operations. 
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is transformed into a vector along the 1-axis 


Pai = (l — 2uu r ) ai = fcei = 


/ k \ 

0 


\V 


Multiplication with the transpose vector gives 


k 1 = (Phi) 1 P a L = a[ P T P&\ = |a 


and 


k = ± |ai| 

can have both signs. From (5.60) we have 
ai — 2u(ua0 = ke 
Multiplication with a[ gives 
2(uai) 2 = |ai| 2 - fe(aiei) 
and since 

|a, - kei | 2 = |ai | 2 + k 2 - 2k(aiei) = 2|ai | 2 - 2 k(a x e { ) 
we have 

2(uai) 2 = ^|at - A'ei| 2 

and from (5.63) we find 

ai — ke i ai — ke i 

u =-= -. 

2uai |ai — /:ei| 

To avoid numerical extinction the sign of k is chosen such that 
cr = sign(k) = -sign(an). 


(5.60) 


(5.61) 


(5.62) 


(5.63) 


(5.64) 


(5.65) 


(5.66) 


(5.67) 


(5.68) 


Then, 
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J 2(«!i + ■ • • + a~ i) + 2\a\\ a\i + ■ ■ ■ + aJ tl 
^sign(an) ^|a n | + yja\ x + a|i + ■ ■ • ^1^ 


^21 


^«1 


(5.69) 


/ sign(an) (|an| + + .. .a^ \ 


021 


( a n + '' ' + a ,7!) + |ailiy«n + ■ ■ ■ + 


= (aii + laillV^f 
V 


+ . . . + ^21 “ 1 “ ' ' ' " 1 “ 


(5.70) 


and the Householder transformation of the first column vector of A gives 
^ —sign( a n )yja 2 n ...a^ 


(1 - 2uu r )ai = 


V 


(5.71) 




Thus after the first step a matrix results of the form 


A (1) = P, A = 


0 CI 22 0-2n 


V 0 ... a™ ) 


In the following (n-2) steps further Householder reflections are applied in the sub¬ 
space k < i, j < n to eliminate the elements 


@-k+l,k ■ - ■ ®n,k 


of the k — th row vector below the diagonal of the matrix: 
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■ 

fl (1) 

■ u \,k-\ 

tf (1) 

“l,Jk * 

• \ 



0 

a <*~ D 

• u k-l,k- 

a k-l,k 


A (k ~ l) = P k _, 

■ ■ P\A = 


0 

a (k ~ l) 

a k,k 

Cl k,n 





u k+l,k 

a (k ~ l) 

a k+l,n 



l o . 

0 

ik- 1) 
a , . 

n,k 

i 

• w n,n / 


( U 

Pk = 

V 


\ 


ljfc-l 

1 - 2uu r / 


Finally an upper triangular matrix results 




(°-u 

0 

a w 
“12 ’ 

(2) 

«22 • 

■ «S-t 

■ «S-t 

« (1) \ 
U l,n \ 

(i) 

a 2,n 

A'”- 1 ) = (/>„_! . 

. Pi)A = R = 


0 





V 0 

0 . 

(n-D 

w n-l,n-l 
0 

“n-l.n 


(5.72) 


(5.73) 


If the orthogonal matrix Q is needed explicitly additional numerical operations are 
necessary to form the product 


Q = (/->„-, ... Pif. 


(5.74) 


5.3 Linear Equations with Tridiagonal Matrix 


Linear equations with the form 


fei.ri + C 1 X 2 = n 


(5.75) 

a,x,_i + bjXi + c,x/ + i = n 

i = 2 •••(«- 1) 

(5.76) 


a„x „-1 + b„x, 


= r, 


(5.77) 
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can be solved very efficiently with a specialized Gaussian elimination method. 2 
They are important for cubic spline interpolation or second derivatives. We begin by 
eliminating a 2 . To that end we multiply the first line with a 2 /b\ and subtract it from 
the first line. The result is the equation 

P 2 X 2 + C 2 X 3 = pi 

(5.78) 

with the abbreviations 


„ , c\ai r\ci 2 

P 2 = b 2 --— pi = r 2 - -—. 

b 1 b 1 

(5.79) 

We iterate this procedure 


/%Xi + C{X i+ 1 = pi 

(5.80) 

0 j Ci~\ai pi — \ cij 

Pi = bj --- pi — Vi --- 

A- 1 Pi -1 

(5.81) 


until we reach the n-th equation, which becomes simply 


— Pn 

(5.82) 

a , Cti-l^n Pn — l^-n 

Pn — bn r. Pn — ^n ^ 

Pn -1 Pn -1 

(5.83) 

Now we immediately have 


Pn 

X " = Jn 

(5.84) 

and backward substitution gives 


Pi — 1 Ci — \Xi 

x i-l = R 

Pi -1 

(5.85) 

and finally 


n - dx 2 

XX ~ P 2 ' 

(5.86) 


This algorithm can be formulated as LU decomposition: Multiplication of the 
matrices 


2 This algorithm is only well behaved if the matrix is diagonal dominant \bj\ > \a, | + |c,-|. 
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L = 


/ 1 \ 
h l 
h i 

V l " 1 / 


gives 


£/ = 


/ A ci \ 

Pi Cl 
A c 3 

v a / 


/A ci 


LU = 


hPi-i A'Cj—i + A) c/ 


\ 


(5.87) 


(5.88) 


V 


InPn —1 OnCn— 1 Ai) / 


which coincides with the matrix 

/A ci \ 

«2 '■ ' • 

fl; bi Cj 

c/} ] b n _ | c n _ | 

A / 



if we choose 



since then from (5.81) 


(5.89) 


(5.90) 


bj — Pi -f lj Ci —i 

and 


(5.91) 


// Pi —l — Cli • 


(5.92) 
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5.4 Cyclic Tridiagonal Systems 


Periodic boundary conditions lead to a small perturbation of the tridiagonal matrix 

(b\ ci a\ \ 

ai ' ■ 


A = 


Qi bi Ci 


(5.93) 


Qn — \ b n — 1 C n — 1 

\ C n t/i: / 

The system of equations 


Ax = r 


(5.94) 


can be reduced to a tridiagonal system [41] with the help of the Sherman-Morrison 
formula [42], which states that if Aq is an invertible matrix and u, v are vectors and 


I+v^AqWo 
then the inverse of the matrix 3 
A = A 0 + uv r 


is given by 


A" 1 



Vuv r A 0 ' 

1 +v r Ao 1 u‘ 


We choose 


uv 


T 


( a \ 

o 

o 

\Cn ) 


[a ci\ \ 


\ Cn 


a\c„ 
a / 


(5.95) 


(5.96) 


(5.97) 


(5.98) 


3 Here uv 3 is the outer or matrix product of the two vectors. 
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Then 


( (bi - a) ci 
02 


A 0 = A - uv r = 


Oj bj Ci 


0 \ 


V 0 


&n —1 


bn — 1 Cn —1 

On (b n ~^)J 


(5.99) 


is tridiagonal. The free parameter a has to be chosen such that the diagonal elements 
do not become too small. We solve the system (5.94) by solving the two tridiagonal 
systems 

A 0 x 0 = r 

A 0 q = u (5.100) 

and compute x from 


x = A = A 0 *r — 


(A 0 - 1 u)v r (Ao 1 r) 
l + v r (Ao‘u) 


Xo - q 


T 

V Xo 


1 


v T q 


(5.101) 


5.5 Linear Stationary Iteration 

Discretized differential equations often lead to systems of equations with large sparse 
matrices, which have to be solved by iterative methods which, starting from an initial 
guess Xo (often simply xo = 0 or Xo = b ) construct a sequence of improved vectors 
by the iteration 

x ( " +| ) = <P(x <n) , b) (5.102) 

which under certain conditions converges to the fixed point 


x F p = <P{x FP ,b) 


(5.103) 


which solves the system of equations 


Ax = b. 


(5.104) 
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A linear iterative method is called stationary, if it has the form 

x (,!+1) = Bx w + Cb (5.105) 

where the matrices B (the so called iteration matrix) and C are constant and do not 
depend on the iteration count n. A fixed point of (5.105) solves (5.104) and hence 
the method is consistent, if 

x—Bx + Cb—Bx + CAx (5.106) 

and hence 

B = I-CA (5.107) 

which brings the iteration to the general form 

x (,,+1) = (/ - CA)x {n] + Cb = x (,,) - C(Ax (n) - b) (5.108) 

r (»+D = (i _ AC)r (n \ (5.109) 

5.5.1 Richardson-Iteration 

The simplest of these methods uses C = col with a damping parameter u>. It iterates 
according to 

x (,!+1) = x ( '° - w(Ax ( '° - b) (5.110) 

r (n+1) = (1 — wA)r w . (5.111) 

The Richardson iteration is not of much practical use. It serves as the prototype 
of a linear stationary method. To improve convergence (5.104) usually has to be 
preconditioned by multiplication with a suitable matrix P 

PAx = Pb (5.112) 

for which the Richardson iteration 

X («+D = x (») _ w p(Ax (n) - b) (5.113) 


is of the general form (5.108). 
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5.5.2 Matrix Splitting Methods 

Let us divide the matrix A into two (non singular) parts [2] 


A = A,+A 2 (5.114) 

where Ai can be easily inverted and rewrite (5.104) as 

Aix = b-A 2 x (5.115) 

which defines the iteration 

<2>(x) = -A7 1 A 2 x + A^b (5.116) 

= -A^\A - Ai)x + A^b = x - A7 [ (Ax - b). (5.117) 


5.5.3 Jacobi Method 


Jacobi divides the matrix A into its diagonal and two triangular matrices [43]: 

A = L + U + D. (5.118) 


For A\ the diagonal part is chosen 


A\ — D 
giving 

x (n+1) = —D~ l (A - D)x (n) + D ~ l b 
which reads explicitly 


r («+t) 

x i 


1 («) 
■— 2^“u x .i 
a 11 m 


H- bj. 

ci a 


(5.119) 


(5.120) 


(5.121) 


This method is stable but converges rather slowly. Reduction of the error by a factor 
of 10~ p needs about iterations. N grid points have to be evaluated in each iteration 
and the method scales with 0(N 2 ) [44], 
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5.5.4 Gauss-Seidel Method 

With 

Ai= D + L 
the iteration becomes 

(D + L)x ( " +1) = — £/x (,,) + b 


(5.122) 


(5.123) 


which has the form of a system of equations with triangular matrix [45], It reads 
explicitly 


z^r +l) =- x + b <- 


j—i 


Forward substitution starting from x\ gives 


(5.124) 


; = 1 : 


i = 2: 


i = 3 : 


r ("+i) 

l i 




(«+l) 
r 2 — 


(«+t) 

*3 — 


ail 

1 

«22 

1 

<733 


1—2 

(b+ 1) («'ii 

-«2i^i - y ajjXj +b 2 

j> 3 


(n+1) (b+1) X ’ (b) 1 / 

-« 31*1 -< 232^2 - 2^, a ‘J x j + ^3 

7>4 


..(n + 1 ) 


~ X a y*j" +1 ’ - X a v*j ,) + b ‘ j • 


(5.125) 


This looks very similar to the Jacobi method. But here all changes are made immedi¬ 
ately. Convergence is slightly better (roughly a factor of 2) and the numerical effort 
is reduced [44]. 


5.5.5 Damping and Successive Over-relaxation 

Convergence can be improved [44] by combining old and new values. Starting from 
the iteration 
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A!X ( ' !+1) = (Ai - A)x (n) + b (5.126) 

we form a linear combination with 

Dx (n+1) = Dx 0,) (5.127) 

giving the new iteration equation 

((1 - uj)D + u;Ai)x ( " +1) = ((1 - u)D + uAi - wA)x , " ) + tub. (5.128) 

In case of the Jacobi method with D — A\ we have 

Dx {n+l) = (D- wA)x w + ujb (5.129) 

x ( " + i) = x (,!) - uD~ l (Ax — b) (5.130) 

which can be also obtained directly from (5.113). 

Explicitly, 

x, (n+1) = (1 - w)x i ( " ) + — | - ’J'aijxf* + b, | . (5.131) 

a “ V & J 

The changes are damped (0 < u> < 1) or exaggerated (1 < u> < 2). 

In case of the Gauss-Seidel method with A\ = D + L the new iteration 4 (5.128) 
is 


(D + wL)x (n+1) = (D + loL — ujA)x (n) + wb = (1 - uj)Dx w - loUx w + ujb 


x («+l) _ x (") _ 



or explicitly 


(5.132) 

(5.133) 


„(«+!) 

x i 


(1-0,)^ + ^ -Z 


>+l) 


ClaX]' ’ ' — ^ 


'J j 


a ijXj 


(") 


(5.134) 


4 This is also known as the method of successive over-relaxation (SOR) and differs from the damped 
Gauss-Seidel method which follows from (5.113). 
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It can be shown that the successive over-relaxation method converges only for 0 < 
u> < 2. For optimal choice of w about { p\f~N iterations are needed to reduce the 
error by a factor of 10 _p . The order of the method is O (/V ?) which is comparable 
to the most efficient matrix inversion methods [44] . 


5.6 Non Stationary Iterative Methods 

Non stationary methods use a more general iteration 

x (,!+1) = 0 n (x n ) (5.135) 

where the iteration function differs from step to step. The method can be formulated 
as a direction search 

X («+D = x (») + AnSn (5.136) 

with direction vectors s„ and step lengths A„. The residual 

r (,!+1) = Ax (,!+1) - b = r (n) + A„As„ (5.137) 

is used as a measure of the remaining error since the exact solution xpp together 
with the error vector 

d <") = x (") _ Xfp (5.138) 

are unknown. Both are, however, related by 

Ad (n) = Ax (n) - Axpp = Ax (,!) - b = r (n) . (5.139) 


5.6.1 Krylov Space Methods 

Solution of the linear system 

Ax = b (5.140) 

can be formulated as a search for the minimum 


min x6R N||Ax - b||. 


(5.141) 
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General iterative methods look for the minimum residual in a subspace of R N 
which is increased at each step. The Richardson iteration, e.g. iterates 

X (" +1) = (1 - wA)x (,,) + wb = x (n) - wr (,,) (5.142) 

r (,!+1) = A(x < " ) - wr (n) ) - b = (1 - uA) r (,,) 

r 0 = Ax 0 - b 

x (1 > = x (0) - wr '°» 

r (1) = r (0) - wAr (0) 

x (2 > = x (1) - ^ = x (0 > - 2wr<® + w 2 Ar <0) 
r® = (1 - W A)r (1 » = r (0) - 2cAr (0 > + w 2 A 2 r® 


Obviously, 

x (n) - x 0 e K n (A, r (0) ) r (n) e K n+l (A, r (0) ) 
with the definition of the n-th Krylov subspace 5 

K n {A, r (0) ) = span{r (0) , Ar (0) , A 2 r (0) , ... A"-'r (0) }. (5.143) 


5.6.2 Minimization Principle for Symmetric Positive Definite 
Systems 


If the matrix A is symmetric and positive definite, we consider the quadratic form 
defined by 


h(x) = Iiq — x r b + ^x r Ax. 
At a local minimum the gradient 
V/z(x) = Ax — b = r 


(5.144) 


(5.145) 


is zero and therefore the minimum of h is also a solution of the linear system of 
equations 


Ax = b. 


(5.146) 


5 For the most common choice xo = 0 we have r <0) = — b and xo + K n (A, r (0) ) = K„(A, r (0) ) = 
K n (A, b). 
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5.6.3 Gradient Method 


The simple Richardson iteration (Sect. 5.5.1 ) uses 


X («+D = x (") _ ur (n) 


with a constant value of u>. 

r (»+l) _ r («) _ wAr («) 


(5.147) 


(5.148) 


Let us now optimize the step width along the direction of the gradient vector. From 
0 = — |r ( " +1) | 2 = r ( " ,r (l - wA)( 1 - u>A) r (,!) 

di jj 

= r (n)r (-2 A + 2u;A 2 )r (n) = -2r (n)T Ar M + 2w|Ar ( '°| 2 (5.149) 

r = 2r l " )7 '(-l +uA)Ar M 


we find the optimum value 

r W T /L r («) 

= r Ar 


|Ar (n) | 2 

The residuals 6 


ro = -b 

r (1) = -b + w (1) Ab 

r (2) = _ b + (w a) + w w )Ab _ u <n u m A 2 \> 


(5.150) 


(5.151) 

(5.152) 

(5.153) 


etc. obviously are in the Krylov subspace 

r (n) e K n+ i(A, b) (5.154) 

and so are the approximate solutions 

x (1) = w (1) b (5.155) 

x (2) = (w (1) + u/ 2) )b - w (2) w (1) Ab (5.156) 

x (n) e K„(A, b). 

6 We assume xq = 0. 


(5.157) 
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5.6.4 Conjugate Gradients Method 


The gradient method gives an approximate solution 

x<»+D = x (0) - u/°¥ 0) - w (1, r (1) - u (n) r (n) 


(5.158) 


but the previously chosen u/ 0) .. . ''are not optimal since the gradient vectors 
r (()) ... r (,1) are not orthogonal. We want to optimize the solution within the space 
spanned by the gradients for which a new basis s (0) ... s (,,, is introduced which will 
be determined later 


K n+ 1 = span(r (0) ... r (,,) ) = span(s (0) ... s (n) ). (5.159) 

Using s (,,) as search direction the iteration becomes 

*("+i) = *(") +A w s ( ") (5.160) 

r (B+ i) = r (B) + A (n) As (n) . (5.161) 

After n+1 steps 

r (»+D _ r (0) + z X U) As Q> = A(d (0) + ^ X U) s U) ). (5.162) 

Multiplication with s (m) gives 

s (m)r r (n+l) _ s (m)T A( j(0) + z X (j) s (m)T As (j) (5.163) 

which, after introduction of the A—scalar product which is defined for a symmetric 
and positive definite matrix A as 

(x,y) A = x r Ay (5.164) 


becomes 

s (m)r r (n+l) _ ^ s (m) ,j( 0 )) A +z A 0 ) (s (m) ,s °')) A 

i =o 

which simplihes considerably if we assume A-orthogonality of the search directions 


(s (m \ s <J) ) = 0 form ^ j 


(5.165) 


because then 
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s (m)r r (n+l) = ( S (m) ) + \(>“)( s ( m ), S (™)) A = s ( m ) r r ( 0) + A (m, S (m)7 'As (m) . 

(5.166) 


If we choose 


A (m) = — 


s (m)r r (0) 
s (m)r As (m) 


(5.167) 


the projection of the new residual r ( " +1) onto K n+ \ vanishes, i.e. this is the optimal 
choice of the parameters A (0) ... X (n) . 

Obviously the first search vector must have the direction of r l0) to span the same 
one-dimensional space. Therefore we begin the iteration with 


s (0) _ r (0) 

(5.168) 

„ |r ( 0 h 2 


\(0) _ 11 

(5.169) 

r (0)r ^ r (0) 

X« = X® + A (°V°>. 

(5.170) 


For the next step we apply Gram-Schmidt orthogonalization 


s 


(i) 


_ r (i) 


„(0) 


(r ( 1 >,sW) A 

(s(°),s<°)) A - 


(5.171) 


For all further steps we have to orthogonalize s ( " + 11 with respect to all of s ( "'... s (0) . 
But, fortunately, the residual r (n+l ’is already A— orthogonal to s (n_1) ... s (0> . This can 
be seen from (5.161) 


r 0+i) _ r U) = \U) As U) (5.172) 

(r (n+1) , s 0) ) A = r ( " +1)r As ( - ,) = J T ^ r ( ' ,+1,7 '(r°' +1) - r U) ). (5.173) 

We already found, that r (,!+1) is orthogonal to K n+ \ , hence to all r (n) ,... r (< ”. 
Therefore we conclude 


( r (n+t), s (j) )a = o for j + 1 < n (5.174) 

Finally we end up with the following procedure 

s (»)r r (0) 

X («+D = x (») + A « s (") with A (») = — (5.175) 

s(") T AsW 

(5.176) 


r ("+l) _ j.(") _j_ A (")yl s (") 
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r (n + l)r Ac(n) 

s («+l) = r («+D _ p(n) s (n) wkh p(n) = (5 . 177) 

As (n) 

This method [25] solves a linear system without storing the matrix A itself. Only 
the product As is needed. In principle the solution is reached after N = dim(A) 
steps, but due to rounding errors more steps can be necessary and the method has to 
be considered as an iterative one. 

The expressions for A and (3 can be brought to numerically more efficient form. 
From (5.162) we find 


s («)L r (0) 



^A^As® j . 

J= 0 / 


(5.178) 


But due to A-orthogonality of the search directions 

s (n)T r (0) _ s (n)T r (n) _ r (n)T r (n) 

which simplifies 

r (n)T r (n) 

\(n) _ F r 

s («)7-^4 S («) • 

Furthermore, from (5.176) and orthogonality of the residual vectors 

r (n+l)r r (n+l) _ ^(n) r (/i + l)7'^ s (/!) 


(5.179) 


(5.180) 


(5.181) 


from which 

r (n+l)7- 4 „(n) _ 1 r («+l)r (n+1) 

fl(«) _ r AS _ A<"> r 

^ s («)7- As (n) g(n)T 

r (n+l)T r (n+l) 

r (n)T r (n) 


(5.182) 


The conjugate gradients method is not useful for non symmetric systems. It can 
be applied to the normal equations (11.32) 


A T Ax = A t b 


(5.183) 


which, for a full-rank non singular matrix have the same solution. The condition 
number (Sect. 5.7), however, is 

cond(A r A) = (condA) 2 (5.184) 


and the problem may be ill conditioned. 
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5.6.5 Non Symmetric Systems 

The general minimum residual method (GMRES) searches directly for the minimum 
of 11 Ax — b 11 in the Krylov spaces of increasing order K„ (A, r (0) ). To avoid problems 
with linear dependency, first an orthogonal basis 

K„(A, r (0) ) = span(q 1 ,q 2 ,.. ,q„) (5.185) 

is constructed with Arnoldi’s method, a variant of Gram-Schmidt orthogonalization. 
Starting from the normalized vector 

r (0) 

* = ^ (5 - 186) 

the dimension is iteratively increased by orthogonalizing the vector Aq„with respect 
to K n (A, r (0) ) 7 


q„+t = Mn ~ q «^ q 7 = Mn ~ X (q J Aq ” )q f = Aq » ~ 

7=1 7=1 7=1 

(5.187) 

and normalizing this vector 

h n+l , n = |q, 1+1 | q„ +1 = -^±L. (5.188) 

hn+\,n 

Then 

n 

Aq /? — ^t«+i,nqn+i + ti jn tj / (5.189) 

7=1 


which explicitly reads 

Aqi = /i 2 iq 2 + Anqi = (qi,q 2 ) 

Aq 2 = /i 32 q3 + Ai 2 qt + /i 22 q 2 = (qt, q 2 , q 3 ) 



(5.190) 

(5.191) 


7 q 2 is a linear combination of qiandAqi, q 3 one of qi, q 2 and Aq 2 hence also of qi, Aqi, 
A 2 qi etc. which proves the validity of (5.185). 
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Aq„ — X; i 1. n *1,: j 1 T hlnQl T - • • h nn f\ n . 
The new basis vectors are orthogonal 8 since 


q[q 2 = qf [Aqi - (qf^qi)qi] = (qfAqoa - |qi| 2 ) = o 

and induction shows for k = 1 ... 77 

n n 

q[q„+i = q[Aq„ - ^(qjAq„)(q[q 7 ) = q[Aq„ - ^(qj Aq„)6 kJ 
j =1 j=i 

We collect the new basis vectors qi ... q„ into a matrix 

U n = (qi,.. .q„) 

and obtain from (5.190) to (5.192) 


AU„ = U n+ \H 


with the (n + 1) x n upper Hessenberg matrix 


/ h\\ h\2 ■ ■ ■ 

hn hii 


H = 


hi2 ' • 


h\n \ 

^2 n 


V 


,L nn 

h n i ]. /; / 


Since 


x (n) -x 0 e K„(A, r (0) ) 


it can be written as a linear combination of qi ... q„ 
x (n) - x 0 = (qi... q„)v. 

The residual becomes 

r w = A(qi... q„)v + Ax 0 - b = A(q, ... q„)v + r <0) 
= U„+iHy+ |r (0) |qi 


(5.192) 


= 0. 


(5.193) 


(5.194) 


(5.195) 


(5.196) 


(5.197) 


if q»+t = 0 the algorithm has to stop and the Krylov space has the full dimension of the matrix. 
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( l \ 



H\+ |r (0) | 

0 


Un-\- 1 





W 



(5.198) 


hence 


r (”)| 2 = 

H\+ |r (0) | 

/!\ 

0 


T 

ut +1 u n+l 

H\+ |r (0) | 

l l \ 

0 




W 







But since 



/ q /\ 

Z 1 

\ 

ut +1 u n+l = 

) 

(q, ... q„) = 

V 


is a n x n unit matrix, we have to minimize 


r w | = 

H\+ |r (0) | 

l l \ 

0 




UJ 



(5.199) 


(5.200) 


(5.201) 


which is a least square problem, since there are n + 1 equations for the n unknown 
components of v. It can be solved efficiently with the help of QR decomposition 
(11.36) 



(5.202) 


with a(n+l)x(n + l) orthogonal matrix Q. The norm of the residual vector 
becomes 


(o) , + |r(0>l 

l l \ 

0 

— 

( fi 0 V ) + |r ,0 >IC r 

/ 1 \ 

0 


U/ 



U/ 


(5.203) 
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Substituting 




|r (n) l =Jy?, +l + \Rv + y\ 2 

which is obviously minimized by solving the triangular system 


(5.204) 


(5.205) 


R\ + y = 0. (5.206) 

The GMRES method usually has to be preconditioned (cf.5.1 12) to improve con¬ 
vergence. Often it is restarted after a small number (e.g.20) of iterations which avoids 
the necessity to store a large orthogonal basis. 


5.7 Matrix Inversion 

LU and QR decomposition can be also used to calculate the inverse of a non singular 
matrix 

AA~ l = 1. (5.207) 

The decomposition is performed once and then the column vectors of A -1 are cal¬ 
culated similar to (5.27) 

L (UA~ l ) = 1 (5.208) 

or (5.40) 

RA~ l = Q\ (5.209) 

Consider now a small variation of the right hand side of (5.2) 

b + rib. (5.210) 

Instead of 


ri _1 b = x 


(5.211) 
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the resulting vector is 

A~'(b + Ab) = x+ Ax (5.212) 

and the deviation can be measured by 9 

l|Ax|| = MA-'llHAbll (5.213) 

and since 


A|| 


b|| 


(5.214) 


the relative error becomes 


l|Ax|| 
I |x| | 


l|A||||A- 1 || 


l|Ab|| 

llbll 


(5.215) 


The relative error of b is multiplied by the condition number for inversion 


cond(A) = 11A11 ||A _1 ||. 


(5.216) 


Problem 

Problem 5.1 (Comparison of different direct Solvers, Fig. 5.2) In this computer 
experiment we solve the system of equations 

Ax = b (5.217) 

with several methods: 

• Gaussian elimination without pivoting (Sect. 5.1), 

• Gaussian elimination with partial pivoting (Sect. 5.1.1), 

• QR decomposition with Householder reflections (Sect. 5.2.2), 

• QR decomposition with Gram-Schmidt orthogonalization (Sect. 5.2.1), 

• QR decomposition with Gram-Schmidt orthogonalization with extra orthogonal- 
ization step (5.55). 

The right hand side is chosen as 


9 The vector norm used here is not necessarily the Euclidean norm. 
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Fig. 5.2 (Comparison of different direct solvers) Gaussian elimination without ( circles ) and with 
(x) pivoting, QR decomposition with Householder reflections (squares), with Gram-Schmidt orthog- 
onalization ( diamonds ) and including extra orthogonalization (+) are compared. The maximum dif¬ 
ference max; = i ... n (|Xj — xf xact \) increases only slightly with the dimension n for the well behaved 
matrix (5.224, a) but quite dramatically for the ill conditioned Hilbert matrix (5.226,b) 


b = A 


( l \ 

2 


w 


hence the exact solution is 


x = 


/ l \ 

2 

\ n ) 


(5.218) 


(5.219) 


Several test matrices can be chosen: 

• Gaussian elimination is theoretically unstable but is stable in many practical cases. 
The instability can be demonstrated with the example [38] 


A = 


/ 1 1 \ 

-1 1 1 

- 1-11 1 


V-1-1-1-! 1/ 


(5.220) 


No pivoting takes place in the LU decomposition of this matrix and the entries in 
the last column double in each step: 
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/I 

1 \ 


/I 

1 \ 



1 \ 

9 


1 

2 


1 

2 


z 

A« = 

-1 1 

2 

A f2 > = 

1 

4 

...A" 1 " D = 


4 


l -1-1 

-12 ) 


l -1 

- 1 4 J 



2 "-l ) 


(5.221) 

Since the machine precision is cm — 2 -53 for double precision calculations we 
have to expect numerical inaccuracy for dimension n > 53. 


• Especially well conditioned are matrices [46] which are symmetric 

Aij = Aji 

and also diagonal dominant 

£|Ay| < l A "|. 

1 ¥* 

We use the matrix 


jn 1 ... 1 1\ 
1 n ... 1 1 


(5.222) 


(5.223) 


A = 


(5.224) 


11 1 n 1 

\11 1 In/ 

which can be inverted explicitly by 


ja b ... b b\ 
b a b b 


A” 1 = 


b b b a b 
\bb b b a/ 


T’ b = ~ 


1 


2n 2 — 3 n + 1 


(5.225) 


and has a condition number 10 which is proportional to the dimension n (Fig. 5.3). 
• The Hilbert matrix [47, 48] 


10 Using the Frobenius norm ||A|| = J^ij Afj. 
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Fig. 5.3 (Condition 
numbers) The condition 
number corn!(A) increases 
only linearly with the 
dimension n for the well 
behaved matrix (5.224, full 
circles ) but exponentially for 
the ill conditioned Hilbert 
matrix (5.226, open circles ) 



l X 

1 

2 

1 

A= 3 


1 1 

2 3 

1 1 

3 4 

1 1 

4 5 


1 

n 

1 

n +1 

1 

n +2 


\ 


i i i J_ 

' n n +1 n +2 


-a^T 


(5.226) 


is especially ill conditioned [49] even for moderate dimension. It is positive definite 
and therefore the inverse matrix exists and even can be written down explicitly 
[50]. Its column vectors are very close to linearly dependent and the condition 
number grows exponentially with its dimension (Fig. 5.3). Numerical errors are 
large for all methods compared (Fig. 5.2). 


• random matrices 


Aij = £e[- 1 , 1 ]. 


(5.227) 







Chapter 6 

Roots and Extremal Points 


In computational physics very often roots of a function, i.e. solutions of an equation 
like 


f{x i - - - JCjv) = 0 (6.1) 

have to be determined. A related problem is the search for local extrema (Fig. 6.1) 
max fix i • • • Xn) min f(x i • • • xn) (6.2) 


which for a smooth function are solutions of the equations 


dfjx i ■ --xn) 
dxi 


i = l... N. 


(6.3) 


In one dimension bisection is a very robust but rather inefficient root finding method. 
If a good starting point close to the root is available and the function smooth enough, 
the Newton-Raphson method converges much faster. Special strategies are necessary 
to find roots of not so well behaved functions or higher order roots. The combination 
of bisection and interpolation like in Dekker’s and Brent's methods provides gener¬ 
ally applicable algorithms. In multidimensions calculation of the Jacobian matrix is 
not always possible and Quasi-Newton methods are a good choice. Whereas local 
extrema can be found as the roots of the gradient, at least in principle, direct optimiza¬ 
tion can be more efficient. In one dimension the ternary search method or Brent’s 
more efficient golden section search method can be used. In multidimensions the 
class of direction set search methods is very popular which includes the methods 
of steepest descent and conjugate gradients, the Newton-Raphson method and, if 
calculation of the full Hessian matrix is too expensive, the Quasi-Newton methods. 


© Springer International Publishing AG 2017 

P.O.J. Scherer, Computational Physics , Graduate Texts in Physics, 
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6 Roots and Extremal Points 


Fig. 6.1 Roots and local 
extrema of a function 



6.1 Root Finding 


If there is exactly one root in the interval < x < ho then one of the following 
methods can be used to locate the position with sufficient accuracy. If there are 
multiple roots, these methods will find one of them and special care has to be taken 
to locate the other roots. 


6.1.1 Bisection 

The simplest method [51] to solve 

fix) = 0 (6.4) 

uses the following algorithm (Fig. 6.2): 

(1) Determine an interval [ao, bo], which contains a sign change of fix). If 
no such interval can be found then f{x) does not have any zero crossings 

(2) Divide the interval into [ao, ao + b °~ a ° ] [qq + , ho] and choose that 

interval [ai, hi], where fix) changes its sign. 

(3) repeat until the width h„ — a n < s is small enough. 1 

The bisection method needs two starting points which bracket a sign change of 
the function. It converges but only slowly since each step reduces the uncertainty by 
a factor of 2. 


'Usually a combination like e = 2 Em + \b n |e r of an absolute and a relative tolerance is taken. 
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Fig. 6.2 Root finding by 
bisection 


Fig. 6.3 Regula falsi 
method 



6.1.2 Regula Falsi (False Position) Method 


The regula falsi [52] method (Fig. 6.3) is similar to the bisection method [51]. 
However, polynomial interpolation is used to divide the interval [x r , a r ] with 
f (x r )f(a r ) < 0. The root of the linear polynomial 

f (a r ) — f{x r ) 

P(X) = f(X r ) + (X- X r ) J (6.5) 

a r — x r 

is given by 


a r — x r a r f (x r ) - x r f (a r ) 

f = x r — t (x r )-=- 

f{a r )-f{x r ) f{x r )-f{a r ) 


( 6 . 6 ) 


which is inside the interval [x r , a, ]. Choose the sub-interval which contains the sign 
change: 
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/(*r)/(£r) < 0 [x r+u a r+l ] = [x r , £ r ] 

f(Xr)f(£r) > 0 -* [Xr+l,a r+ i\ = [6, «r]- (6.7) 

Then £, provides a series of approximations with increasing precision to the root of 
fix) = 0. 


6.1.3 Newton-Raphson Method 


Consider a function which is differentiable at least two times around the root £. 
Taylor series expansion around a point xq in the vicinity 


fix) = fix o) + (x - xo)/'(xo) + -(x - x 0 ) 2 /"(xo) + 
gives for x = £ 


( 6 . 8 ) 


0 = /(x 0 ) + (£ - x 0 )f'ix 0 ) + -i£ - x 0 ) 2 /"(x 0 ) H-. 


(6.9) 


Truncation of the series and solving for £ gives the first order Newton-Raphson 
[51, 53] method (Fig. 6.4) 


Xr+l — x r 


fiXr) 


f'iXr) 

and the second order Newton-Raphson method (Fig. 6.4) 
f'ix r ) ± sjf'ix,.) 1 ~ 2 fix r ) f'iXr) 


X r _)_i — x r 


f'iXr) 


( 6 . 10 ) 


( 6 . 11 ) 


Fig. 6.4 Newton-Raphson 
method 
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Fig. 6.5 Secant method 



The Newton-Raphson method converges fast if the starting point is close enough 
to the root. Analytic derivatives are needed. It may fail if two or more roots are 
close by. 


6.1.4 Secant Method 


Replacing the derivative in the first order Newton Raphson method by a finite dif¬ 
ference quotient gives the secant method [51] (Fig. 6.5) which has been known for 
thousands of years before [54] 


x r +i = x r - f{x r ) 


■ X r —\ 


f(x r ) - 


( 6 . 12 ) 


Round-off errors can become important as \f(x r ) — /(x,_i)| gets small. At the 
beginning choose a starting point x 0 and determine 


Xl = x 0 - f(x 0 )- 


2h 


f(x o + h) - f(x o - h) 
using a symmetrical difference quotient. 


(6.13) 


6.1.5 Interpolation 

The secant method is also obtained by linear interpolation 

. . X~X r X ~ X r _l 

p(x) = - Jr —l H- Jr- 


X r -\ — x r 


Xr ~ X r —l 


(6.14) 
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The root of the polynomial p(x r+ \ ) = 0 determines the next iterate x r+ i 

*r+ 1 = -- - —(Xrfr- 1 ~ *r-l f r ) = X r ~ f r ^ -. (6.15) 

Jr-1 ~ Jr Jr ~ Jr -1 

Quadratic interpolation of three function values is known as Muller’s method [55]. 
Newton’s form of the interpolating polynomial is 


p(x) = f- + (x - x r )f[x r , x r _i] + (x - x r )(x - x r -i)f[x r , x r -i,x r - 2 ] (6.16) 


which can be rewritten 


p(x) = fr + (x - X r )f[x r , X r _l] + (X - x r ) 2 f[x r ,x r - 1, X,- 2 ] 

+ (x r - x r -l){x - Xr)f[x r ,X r -l,X r - 2 \ 

= f r + (X - X r ) 2 f[x r , X r -l,X r -2] + (x ~ X r ) (f[x r ,X r - 1] + f[x r ,X r - 2 ] ~ f[x r - l.X r _ 2 ]) 
= fr + MX ~ X r ) + B(X - X r ) 2 ( 6 . 17 ) 


and has the roots 


A , r ^ 2 Jr 

X.r -\-1 Xp _L \ - 

+ 2B V 4 B 2 B 


(6.18) 


To avoid numerical cancellation, this is rewritten 

x r+ i = x r + ^ ± JA 2 - 4B/ r ) 

= x r + —z--fA =p ^A 2 - 4/7/; ) 

A 2 — (A 2 — ABf r ) V Jr ) 

, - 2 /,. 

'' A ± v/A 2 - 4B/ ( .' 


(6.19) 


The sign in the denominator is chosen such that x r+ \ is the root closer to x r . The 
roots of the polynomial can become complex valued and therefore this method is 
useful to find complex roots. 


6.1.6 Inverse Interpolation 

Complex values of x r can be avoided by interpolation of the inverse function 
instead 


x = r\y). 


( 6 . 20 ) 
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Fig. 6.6 Root finding by 
interpolation of the inverse 
function 



Using the two points x r , x r -\ the Lagrange method gives 


p(y) = X r -1 


y - fr 
fr-l ~ fr 


+ X r 


y - fr -1 

fr ~ fr-l 


( 6 . 21 ) 


and the next approximation of the root corresponds again to the secant 
method (6.12) 


X r-lfr ~ Xrfr-l . (x r -l ~ X r ) 

x r+1 = p( 0) = -----= X,. + —--- /,.- 


fr - fr- 


fr - fr-l 


( 6 . 22 ) 


Inverse quadratic interpolation needs three starting points x r , x>._. i, x ,-_2 together 
with the function values f r , f r -i, f r -1 (Fig- 6.6). The inverse function x = f~ l (y) 
is interpolated with the Lagrange method 


, , (y - fr-l)(y - fr) , (y - fr-l){y - fr) 

(fr-2 ~ fr — l)( fr—2 ~ fr) (fr-l ~ fr-l)(fr-l ~ fr) 

(y - fr-l)(y - fr- 2 ) . 

(fr - fr—l)(fr ~ fr-i.S'" 

For y = 0 we find the next iterate 

fr-lfr 


(6.23) 


x r+1 = p( 0) 

■ 


~X r -2 + 


fr—2 fr 


(fr—2 fr — l)(fr—2 fr) (fr-l ~ fr-l)(fr-l ~ fr) 

fr—l fr—2 


X r -l 


(fr ~ fr—l)(fr ~ fr-i) 


X r - 


(6.24) 
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Fig. 6.7 (Validity of inverse quadratic interpolation) Inverse quadratic interpolation is only applica¬ 
ble if the interpolating polynomial p(y ) is monotonous in the range of the interpolated function 
values fi ... f}. (a) and (b) show the limiting cases where the polynomial has a horizontal tangent 
at f\ or fj. (c) shows the case where the extremum of the parabola is inside the interpolation range 
and interpolation is not feasible 


Inverse quadratic interpolation is only a good approximation if the interpolating 
parabola is single valued and hence if it is a monotonous function in the range of 
/ r , fr —\. fr —2 ■ For the following discussion we assume that the three values of x are 
renamed such that xi < xo < x 3 . 

Consider the limiting case (a) in Fig. 6.7 where the polynomial has a horizontal 
tangent at y = / 3 and can be written as 


p(y) = *3 + (x\ - x 3 ) 


(y - h ) 2 

(/i-/ 3 ) 2 ' 


(6.25) 


Its value at y = 0 is 


p(0 ) = x 3 + (xi - x 3 ) 




(/t - f 3 y 


= X 1 + (x 3 - Xi) 


(l _ fi—) 

V (/ 1 -/ 3 ) 2 / 


(6.26) 


If /1 and / 3 have different sign and \f\\ < |/ 3 | (Sect. 6. 1.7.2) we find 

1 _ 

(fi - h) 2 4 


(6.27) 


Brent [56] used this as a criterion for the applicability of the inverse quadratic 
interpolation. However, this does not include all possible cases where interpolation is 
applicable. Chandrupatla [57] gave a more general discussion. The limiting condition 
is that the polynomial p(y) has a horizontal tangent at one of the boundaries X| 3 . 
The derivative values are 
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dp, = f)= x hh ~ h) *3(/i - fi) *i XI 

dy } (fi - /i)(/2 - fi) (/a - /i)(/3 - fi) fi-h fi-fi 

(6.28) 

(/2-/l) f *2(/3-/l) 2 ^l(/3-/l) 2 -^l(/2-/l) 2 ' 

(/3 - /l)(/3 - fl) [ (/2 - /l) 2 ^ (/2 - /l) 2 

= (fl - h)i.x 2 -xi) r _ 2 _ n 
(73 - /l)(/3 - /2) L ' J 

^, x 2(f3 ~ /l) ^1 (73 - /2) -*3 *3 

</v ' " (fl~ fl)(fl~ fi) (fl - fl)(fl - fi) fi Jl fi-fl 

(6.29) 


(fi - fi) r x i(fj - h) 2 (^ - fi) 2 (h - fi) 1 

(fi - fi)(fi - fi) [ (fi - fl) 2 X3 (fi - fl) 2 X3 (fi - fl) 2 

(fi - fi)(xj - xi) r/_i _\ 2 _ i 
(fi - fi)(fi - fi) [U-1/ l-£_ 


with [57] 


£ = 


X 2 - Xi 
X 3 — XI 


0 = 


fl-fl 
.fi - fl 


= „_ 1 = 

X 3 — X\ 


h-h 

h-h 


(6.30) 

(6.31) 


Since for a parabola either f\ < fi < fj or f\ > fi> h the conditions for applica¬ 
bility of inverse interpolation finally become 


<Z> 2 < £ (6.32) 

l-£>(l-<£) 2 (6.33) 

which can be combined into 

m <V^ (6.34) 


This method is usually used in combination with other methods (Sect. 6.1 .7.2). 


6.1.7 Combined Methods 

Bisection converges slowly. The interpolation methods converge faster but are less 
reliable. The combination of both gives methods which are reliable and converge 
faster than pure bisection. 
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Fig. 6.8 Dekker’s method 



6.1.7.1 Dekker’s Method 


Dekker’s method [58, 59] combines bisection and secant method. The root is brack¬ 
eted by intervals [c r , b r ] with decreasing width where b r is the best approximation to 
the root found and c r is an earlier guess for which f(c r ) and f(b r ) have different sign. 
First an attempt is made to use linear interpolation between the points ( b r , f(b r )) 
and ( a r , f(a r )) where a, is usually the preceding approximation a r — /?, _i and is 
set to the second interval boundary a r = c r _i if the last iteration did not lower the 
function value (Fig. 6.8). 

Starting from an initial interval [xq,x\] with sign(/(.ro)) ^ sign(/(xi)) the 
method proceeds as follows [59]: 

initialization 

f\ = fix i) /o = fix o) 
if l/il < l/ol then { 
b = x i c — a = x o 

fb = fl fc = fa = fo) 

else{ 

b = x o c — a = x i 
fb = fo fc = fa = fl] 


iteration 


x s = b - 




c+b 

2 


r b—a 

fb h =A 


If x s is very close to the last b then increase the distance to avoid too small steps 
else choose x s if it is between b and x m , otherwise choose x m (thus choosing the 
smaller interval) 


b + (5sign(c — b ) if abs(x s — b) < S 
x s if b + S < x s < x m or b — 6 > > x m 

x m else 


x. 
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Determine Xk as the latest of the previous iterates xq ... x r -\ for which 
sign(/Ut)) / sign(/(*,.))■ 

If the new function value is lower update the approximation to the root 


fr = f(Xr) 

if \fr\ < \fk\ then { 

a = b b = x r c — Xk 

fa = fb fb= fr fc = fk } 

otherwise keep the old approximation and update the second interval boundary 

if \fr\ > | fk I then { 
b = Xk a = c = x r 

fb = fk fa = fc= fr} 

repeat until \c — b\ < e or f r — 0. 


6.1.7.2 Brent’s Method 


In certain cases Dekker’s method converges very slowly making many small steps of 
the order e. Brent [56, 59, 60] introduced some modifications to reduce such problems 
and tried to speed up convergence by the use of inverse quadratic interpolation 
(Sect. 6.1.6). To avoid numerical problems the iterate (6.24) is written with the help 
of a quotient 


Xr +1 = 

+ 


fbfc 


{fa - fbXfa ~ fc) 
.fb fa 


(2 -|- 


fafc 


{fb ~ fa) {fb ~ fc) 


{fc ~ fb){fc - fa) 

P 

q 


= b+ p - 


with 


(6.35) 


, u , fbifa - fb) n , fb(fb - fc) 

= ( C ~ h ) - 72 -(*-«)- T1 - 

Jc JaJc 

= afbfcifb -fc) + b [fa fb {fb - fa) + fbfcifc ~ fb)] + cf a f„{f a ~ fb) 

fafc 2 


(6.36) 
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q = j\ = (/a - fc)(fb - fc)(fb - fa) 

\ fc / \fc ) \ fa ) fa fc 

(6.37) 

If only two points are available, linear interpolation is used. The iterate (6.22) 
then is written as 

(a — b ) p 

x r+l =b+- - frf b = b+‘- (6.38) 

fb - fa q 

with 

p = {a-b )^ q=('T~ i)- ( 6 - 39 ) 

J a \ fa / 

The division is only performed if interpolation is appropriate and division by zero 
cannot happen. Brent’s method is fast and robust at the same time. It is often recom¬ 
mended by text books. The algorithm is summarized in the following [61]. 

Start with an initial interval [xo, X\] with f(x o) f(x\) < 0 

initialization 

a = x o b — x i c = a 

fa = f(a ) fb = f(b) f c = f a 

e = d = b — a 

iteration 

If c is a better approximation than b exchange values 

if l/cl < \fb\ then{ 
a = b b = c c = a 

fa = fb fb= fc fc = fa) 


calculate midpoint relative to b 


x m = 0.5(c - b) 


stop if accuracy is sufficient 
if \x m | < e or fb = 0 then exit 

use bisection if the previous step width e was too small or the last step did not improve 

if |e| < £ or |/J < |/i| then{ 
e = d = x m } 
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otherwise try interpolation 


else { 

if a = c then { 

r, fb fb ~ fa 

P = 2x m — q = --- 

Ja Ja 

else { 


P = 2x, 

q 


fbifa ~ fb) , fbifb - fc) 

- — (b — a)- 


fc 2 


fafc 


make p a positive quantity 

if p > 0 then {q = —q) else {p = —p) 


update previous step width 
s = e e = d 

use interpolation if applicable, otherwise use bisection 


if2p < 3 x,„q — \eq\ and p < |0.5s^| then{ 



else{e — d = x m } 
a = b f a = fb 


if \d\ > e then { 

b = b + d] 

else {b = b + esign (x m )} 

calculate new function value 
fb = f(b) 

be sure to bracket the root 
if sign(/ fc ) = sign(/ c ) then { 
c = a f c = f a 
e — d — b — a] 


6.1.7.3 Chandrupatla’s Method 

In 1997 Chandrupatla [57] published a method which tries to use inverse quadratic 
interpolation whenever possible according to (6.34). He calculates the relative posi¬ 
tion of the new iterate as (Fig. 6.9). 

x — c 

t = - 

b — c 
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Fig. 6.9 Chandrupatla’s 
method 


f 



fcfb 


b~C (f a - fb)(fa~ fc) 
fc fb + 


a + 


fa fc 


(.fb ~ fa)(fb - fc) 
fa fc 


b + 


fbfa 


ifc - fbXfc ~ fa) 


b-C f c - fa fb~ fa {fb ~ fa){fb - fc) 


(6.40) 


The algorithm proceeds as follows: 

Start with an initial interval [xo, X|] with /(xo)/(xi) < 0. 

initialization 

b = x o a — c = x i 

fb = f(b) f a = f c = f(c) 

t = 0.5 

iteration 

x, — a + t (b — a) 

f = fix,) 

if sign(/,) = sign (/„){ 
c = a fc = fa 
a = X, fa = F,} 
elsej 

c = b b = a a = x, 

fc = fb fb = fa fa = //} 

X,n = bl f„ = f iL 

if abs ( f b ) < abs(/ a ){ 

X,n = b f m = f ,->} 

tol = 2e M \x m \ + e a 
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if // > 0.5 or f m — 0 exit 

£= <P= £=£ 

s c-b _ fc-fb 

if i - < 0 < vcf 


—/t!— 


fc 


+ 


c—a fa 


fb 


fb-fa fb-fc b-a f c -f a f c -fb 1 

else {t = 0.5} 

if t < t/{t — t /} 

if t > (1 — t,){t = 1 — ?/} 


Chandrupatla’s method is more efficient than Dekker’s and Brent’s, especially for 
higher order roots (Figs. 6.10, 6.11 and 6.12). 


6.1.8 Multidimensional Root Finding 

The Newton-Raphson method can be easily generalized for functions of more than 
one variable. We search for the solution of a system of n nonlinear equations in n 
variables x, 


( /l (x\ ■ ■ ■ x n ) \ 

1 = 0 . 

fn(x 1 ' ■■Xn) ) 


(6.41) 


The first order Newton-Raphson method results from linearization of 



Fig. 6.10 (Comparison of different solvers) The root of the equation f{x) = x 2 — 2 is determined 
with different methods: Newton-Raphson (a) ( black squares), Chandrupatla (b) ( indigo circles), 
Brent (c) (red triangles up), Dekker (d) ( green diamonds), regula falsi (e) (blue stars), pure bisection 
(f) (black dots). Starting values are xi = —1, X 2 = 2. The absolute error is shown as function of 
the number of iterations. For x\ = — 1. the Newton-Raphson method converges against — s/2 
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Fig. 6.11 (Comparison of different solvers for a third order root) The root of the equation f(x) = 
(x — l) 3 is determined with different methods: Newton-Raphson (a) (magenta), Chandrupatla 
(b) (orange), Brent (c) (blue), Dekker (d) (red), regula falsi (e) (black), pure bisection (f) (green). 
Starting values are x\ = 0, X 2 = 1.8. The absolute error is shown as function of the number of 
iterations 



Fig. 6.12 (Comparison of different solvers for a high order root) The root of the equation f(x) = 
x 25 is determined with different methods: Newton-Raphson (a) (orange), Chandrupatla (b) (blue 
circles), Brent (c) (black), Dekker (d) (green), regula falsi (e) (magenta), pure bisection (f) (red 
dots). Starting values are xi = — 1, X 2 = 2. The absolute error is shown as function of the number 
of iterations 


0 = f(x) = f(x°) + /(x°)(x - x°) + ■ ■ • 


(6.42) 


with the Jacobian matrix 



9fn ... dU 
dx\ dx n 


(6.43) 
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If the Jacobian matrix is not singular the equation 

0 = f(x°) + /(x°)(x — x°) (6.44) 

can be solved by 

x = x°- (6.45) 

This can be repeated iteratively 

x (r+ D = x (r) - (i(x (r) ))-'f(x (r) ). (6.46) 

6.1.9 Quasi-Newton Methods 

Calculation of the Jacobian matrix can be very time consuming. Quasi-Newton meth¬ 
ods use instead an approximation to the Jacobian which is updated during each 
iteration. Defining the differences 

d w = x (r+1) - x (r) (6.47) 

y (r) = f(x (r+1) ) - f(x (r) ) (6.48) 

we obtain from the truncated Taylor series 

f(x (,+1) ) = f(x (r) ) + /(x (r) )(x (r+1) - x (r) ) (6.49) 

the so called Quasi-Newton or secant condition 

y (r) = /(x ( ' , )d ( ' ) . (6.50) 

We attempt to construct a family of successive approximation matrices J r so that, if 
J were a constant, the procedure would become consistent with the quasi-Newton 
condition. Then for the new update J r+ \ we have 

T r+1 d (r) = y (r) . (6.51) 

Since d ( ' \ y (r) are already known, these are only n equations for the n 2 elements of 
J r+ 1 - To specify J r+ \ uniquely, additional conditions are required. For instance, it is 
reasonable to assume, that 

7,+iU = J r u for all u _L d (,) . (6.52) 


Then J r+ \ differs from ./, only by a rank one updating matrix 
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Jr +1 = Jr +ud (r)T . 

From the secant condition we obtain 
y / - + id (r> = /,d (r) + u(d (r) d (r)r ) = y (r) 
hence 


1 


u = 


(y (r) - / r d w ). 


|JW | 2 * 

This gives Broyden’s update formula [62] 

J,+1 = Jr + < jW - ■ , ' dW ) d<rW - 

To update the inverse Jacobian matrix, the Sherman-Morrison formula [42] 

T , , A _1 uv r A _1 

(A + uv 7 )" 1 = A’ 1 - t 

can be applied to have 

_ 1 _ (y (r> “ Jrd (r) ) d^ r)T J- 1 

J r +1 — J r 


(6.53) 


(6.54) 


(6.55) 


(6.56) 


(6.57) 


= j: 


1 + ^pdwry-i ( y w _ /rd W) 

1 (j-'yW -d M )d M ' / J-' 

d^vrV^ ' 


(6.58) 


6.2 Function Minimization 


Minimization or maximization of a function 2 is a fundamental task in numerical 
mathematics and closely related to root finding. If the function f(x) is continuously 
differentiable then at the extremal points the derivative is zero 

df 

— = 0. (6.59) 

d.r 

Hence, in principle root finding methods can be applied to locate local extrema of 
a function. However, in some cases the derivative cannot be easily calculated or the 


2 In the following we consider only a minimum since a maximum could be found as the minimum 

of-/(*). 
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function even is not differentiable. Then derivative free methods similar to bisection 
for root finding have to be used. 


6.2.1 The Ternary Search Method 

Ternary search is a simple method to determine the minimum of a unimodal function 
/(x). Initially we have to find an interval [ao, bo] which is certain to contain the 
minimum. Then the interval is divided into three equal parts [ao, t'o], [co, do], [do, bo] 
and either the first or the last of the three intervals is excluded (Fig. 6.13). The 
procedure is repeated with the remaining interval [ai, b\\ = [ao, do] or [aj, b\] = 
[co, bo\. 

Each step needs two function evaluations and reduces the interval width by a 
factor of 2 /3 until the maximum possible precision is obtained. It can be determined 
by considering a differentiable function which can be expanded around the minimum 

xq as 


(x — xn) 2 „ 

f(x) = f(x 0 ) + 2 fix o) + • • • . 


(6.60) 


Numerically calculated function values fix) and fix o) only differ, if 


~~ f" (xo) > £ m/Oo) 


(6.61) 


which limits the possible numerical accuracy to 


Fig. 6.13 Ternary search 
method 


f(x) 



a 


c 


d 


x 

b 
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Fig. 6.14 (Ternary search method) The minimum of the function f(x) = 1 + 0.01 jr 2 + 0.1 x 4 is 
determined with the ternary search method. Each iteration needs two function evaluations. After 
50 iterations the function minimum / m j n = 1 is reached to machine precision em ^ 10 -16 . The 
position of the minimum x m ; n cannot be determined with higher precision than s /em & 10~ 8 (6.63) 


e(xo) = min|x — xq| 


2f(x 0 ) 

- £m 

f"(x o) M 


(6.62) 


and for reasonably well behaved functions (Fig. 6. 14) we have the rule of thumb [63] 
e(vo) « y/eM- (6.63) 


However, it may be impossible to reach even this precision, if the quadratic term 
of the Taylor series vanishes (Fig. 6. 15). 

The algorithm can be formulated as follows: 

iteration 


ii(b — a) < 6 then exit 
c = a + ^(b — a) d — a + |(i> — a) 
f c = m f d = f(d ) 

if f c < fd then b — d else a = c 


6.2.2 The Golden Section Search Method (Brent’s Method) 

To bracket a local minimum of a unimodal function f(x ) three points a, h. c are 
necessary (Fig. 6.16) with 


/(«) > f(b) f(c)>f(b). 


(6.64) 
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Fig. 6.15 (Ternary search method for a higher order minimum) The minimum of the function 
/(x) = 1 + 0.1 x 4 is determined with the ternary search method. Each iteration needs two function 
evaluations. After 30 iterations the function minimum / m j n = 1 is reached to machine precision 
Em ^ 10~ 16 . The position of the minimum x m j n cannot be determined with higher precision than 

yeM ** 10- 4 



Fig. 6.16 (Golden section search method) A local minimum of the function f(x) is bracketed by 
three points a , b, c. To reduce the uncertainty of the minimum position a new point £ is chosen in 
the interval a < £ < c and either a or c is dropped according to the relation of the function values. 
For the example shown a has to be replaced by £ 


The position of the minimum can be determined iteratively by choosing a new 
value £ in the interval a <£< c and dropping either a or c, depending on the ratio 
of the function values. A reasonable choice for £ can be found as follows (Fig. 6.17) 
[63, 64]. Let us denote the relative positions of the middle point and the trial point as 


b — a 


c — a 
£ — a 


= P 
£ 



1-/3 


b — 
c — 


— b + b — a 


— t + /3. 


a 

~b 


P S-b 

1 — (3 c — a 


c — a 


c — a 


(6.65) 
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Fig. 6.17 Golden section search method 


The relative width of the new interval will be 


c — a 
£ — a 
c — a 


= (1-/3 —0 

= (t + (3) or 


b — a 

or -= (3 if a < £ < b 

c — a 

c — b 

-= (1 — /?) if b < £ < c. 

c — a 


The golden search method requires that 


t 


1-2 /3 = 


c + a — 2b 
c — a 


(c — b) — (b — a) 
c — a 


( 6 . 66 ) 

(6.67) 


( 6 . 68 ) 


Otherwise it would be possible that the larger interval width is selected many times 
slowing down the convergence. The value of t is positive if c — b > b — a and 
negative if c — b < b — a, hence the trial point always is in the larger of the two 
intervals. In addition the golden search method requires that the ratio of the spacing 
remains constant. Therefore we set 


0 

1-/3 

0 

1-/3 


t + 0 
t 
t 

1-0-t 


1-/3 

t 


if a < t; < b 


t 

0 


if b < £ < c. 


(6.69) 

(6.70) 


Eliminating t we obtain for a < £ < b the equation 
^^(/3 2 + /3 — 1) = 0. 


(6.71) 
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Besides the trivial solution (3 = 1 there is only one positive solution 

V5 - 1 

/? = -- & 0.618. 

2 

For b < £ < c we end up with 
^ t (/ 3 2 - 3 / 3 + 1 ) = 0 
which has the nontrivial positive solution 


(6.72) 


(6.73) 


0 = 


3-V5 

2 


0.382. 


(6.74) 


Hence the lengths of the two intervals [a, b], [b, c] have to be in the golden ratio 
tp = 1+ 2 ~ 1.618 which gives the method its name. Using the golden ratio the 

width of the interval bracketing the minimum reduces by a factor of 0.618 (Figs. 6.18 
and 6.19). 



Fig. 6.18 (Golden section search method) The minimum of the function f(x) = 1 + 0.01 x 2 + 
0.1 x 4 is determined with the golden section search method. Each iteration needs only one function 
evaluation. After 40 iterations the function minimum / m ; n = 1 is reached to machine precision 
Em ^ 10“ 16 . The position of the minimum x mm cannot be determined to higher precision than 
10~ 8 (6.63) 
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Fig. 6.19 (Golden section search for a higher order minimum) The minimum of the function 
f(x) = 1 + 0.1 x 4 is determined with the golden section search method. Each iteration needs only 
one function evaluation. After 28 iterations the function minimum / m ; n = 1 is reached to machine 
precision Em ^ 10~ 16 . The position of the minimum x m j n cannot be determined to higher precision 
than ^ 10~ 4 


The algorithm can be formulated as follows: 

if c — a <5 then exit 
if (b — a) > (c — b) then { 
x = 0.618 Z? + 0.382a 
fx = fix) 

if fx < fb then [c = b b = x f c = fb fb = fx} 
else a — x f a = f x ) 
if (b — a) < (c — b ) then { 
x = 0.618/7 + 0.382c 

fx = fix) 

if fx < fb then [a — b b = x f a = fb fb = fx} 
else c = x f c = f x } 

To start the method we need three initial points which can be found by Brent’s 
exponential search method (Fig. 6.20). Begin with three points 


«0; bo = ao + h, Co + 1.618 h (6.75) 

where /?o is a suitable initial step width which depends on the problem. If the minimum 
is not already bracketed then if necessary exchange ao and b o to have 

/(ao) > fib 0 ) > fic 0 ). (6.76) 


Then replace the three points by 
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Fig. 6.20 Brent’s 
exponential search 


f(x) 



0 u 0 u 0 
®1 ^ C, 


a 2 b 2 c 2 

a 3 b 3 c 3 


ai=bo b\ — co ci = co + 1.618 (co — bo) (6.77) 

and repeat this step until 

f(b n ) < f(c n ) (6.78) 

or n exceeds a given maximum number. In this case no minimum can be found and 
we should check if the initial step width was too large. 

Brent’s method can be improved by making use of derivatives and by combining 
the golden section search with parabolic interpolation [63]. 


6.2.3 Minimization in Multidimensions 

We search for local minima (or maxima) of a function 

h(x) 

which is at least two times differentiable. In the following we denote the gradient 
vector by 



and the matrix of second derivatives (Hessian) by 



(6.80) 
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Fig. 6.21 (Direction set 
minimization) Starting from 
an initial guess Xo a local 
minimum is approached by 
making steps along a set of 
direction vectors s r 


y 

a 



The very popular class of direction set methods proceeds as follows (Fig. 6.21). 
Starting from an initial guess xo a set of direction vectors s r and step lengths \ r is 
determined such that the series of vectors 

x r+i = x r + A r s r (6.81) 

approaches the minimum of h(x). The method stops if the norm of the gradient 
becomes sufficiently small or if no lower function value can be found. 


6.2.4 Steepest Descent Method 

The simplest choice, which is known as the method of gradient descent or steepest 
descent 3 is to go in the direction of the negative gradient 

s r = -g r (6.82) 

and to determine the step length by minimizing h along this direction 


h( A) = li(x r — Ag,.) = min. 


(6.83) 


Obviously two consecutive steps are orthogonal to each other since 


° = 5\ /j(X, ' +1 ~ Ag, ' )|A=0 = “ S /+ 1 &" 


(6.84) 


3 Which should not be confused with the method of steepest descent for the approximate calculation 
of integrals. 
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Fig. 6.22 (Function 
minimization) The minimum 
of the Rosenbrock 
function h(x, y) = 

100(y -x 2 ) 2 + (1 -x) 2 is 
determined with different 
methods. Conjugate (CG) 
gradients converge much 
faster than steepest descent 
(SD). Starting at 
(x,y) = (0,2), 
Newton-Raphson (NR) 
reaches the minimum at 
x = y = 1 within only 5 
iterations to machine 
precision 



number of iterations 




This can lead to a zig-zagging behavior and a very slow convergence of this method 
(Figs. 6.22 and 6.23). 
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Fig. 6.23 (Minimization of the Rosenbrock function) Left Newton-Raphson finds the minimum 
after 5 steps within machine precision. Middle conjugate gradients reduce the gradient to 4 x 10~ 14 
after 265 steps. Right The steepest descent method needs 20000 steps to reduce the gradient to 
5 x 10~ 14 . Red lines show the minimization pathway. Colored areas indicate the function value 
{light blue <0.1, grey 0.1. ..0.5, green 5...50 ,pink > 100). Screen shots taken from problem 6.2 


6.2.5 Conjugate Gradient Method 

This method is similar to the steepest descent method but the search direction is 
iterated according to 


So = -go 

x r+1 = X,. + \ r s r 

Sr+1 = —g/~t~l + Pr+lSr 


(6.85) 

( 6 . 86 ) 
(6.87) 


where A,, is chosen to minimize h(x r+ \) and the simplest choice for 3 is made by 
Fletcher and Rieves [65] 

Pr+l = (6-88) 

9r 

This method was devised to minimize a quadratic function and to solve the related 
system of linear equations, but it is also very efficient for more complicated functions 
(Sect. 5.6.4). 


6.2.6 Newton-Raphson Method 

The first order Newton-Raphson method uses the iteration 


x r+1 = x r - H(x r ) 1 g(x ; .). 


(6.89) 
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The search direction is 

s=//-'g (6.90) 

and the step length is A = 1. This method converges fast if the starting point is close 
to the minimum. However, calculation of the Hessian can be very time consuming 
(Fig. 6.22). 


6.2.7 Quasi-Newton Methods 

Calculation of the full Hessian matrix as needed for the Newton-Raphson method 
can be very time consuming. Quasi-Newton methods use instead an approximation 
to the Hessian which is updated during each iteration. From the Taylor series 

h(x) = h 0 + b T x + ^x T Hx + ■ ■ ■ (6.91) 


we obtain the gradient 


g(x,.) = b + Hx r H -= g(x,._!) + H(x r - x,-_i) H-. (6.92) 


Defining the differences 


d,. = x r+1 - x,. (6.93) 

y r = gr+l - gr (6.94) 

and neglecting higher order terms we obtain the quasi-Newton or secant condition 

Hd r =y r . (6.95) 

We want to approximate the true Hessian by a series of matrices H, which are updated 
during each iteration to sum up all the information gathered so far. Since the Hessian 
is symmetric and positive definite, this also has to be demanded for the II, . 4 This 
cannot be achieved by a rank one update matrix. Popular methods use a symmetric 
rank two update of the form 

H r+ 1 = H r + auu r + f3\\ T . (6.96) 

The Quasi-Newton condition then gives 


H r+ id r = H r d, + a(u r d,)u + /3(v r d,)v = y r 


4 This is a major difference to the Quasi Newton methods for root finding (6.1.9). 


(6.97) 
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hence H, d,- — y r must be a linear combination of u and v. Making the simple choice 
u = y r v = H, d, (6.98) 

and assuming that these two vectors are linearly independent, we find 


P = 


1 


1 


(v r d,.) (dj’//,d,) 

1 1 


(u r d,.) (yjTd,) 


(6.99) 

( 6 . 100 ) 


which together defines the very popular BFGS (Broyden, Fletcher, Goldfarb, Shanno) 
method [66-69] 


,, , y r y T r (H r d r )(H r d r ) T 

H r+ i = H r H-—- -. 

y; d, d; H, d, 


( 6 . 101 ) 


Alternatively the DFP method by Davidon, Fletcher and Powell, directly updates the 
inverse Hessian matrix B = H according to 


B ,+1 = B, 


d,d, r (B r y r )(B r y r ) 3 


yjd r yj B, y, 


( 6 . 102 ) 


Both of these methods can be inverted with the help of the Sherman-Morrison 
formula to give 


(d r - B r y r )dj + d, (d, - By r ) T (d,. - B r y r ) T y T 
B r+ 1 = B r H- r —^ - / 7 . 2 - dd' (6.103) 


y, r d,- 


(yjd) 


(y,. - H r d r )y T r + y r (y r - H r d,.) T (y r - H r d,)d,. T 

H '+ 1 = Hr + -^-H \- y ' y '- ' 


y, r d,- 


(y r r d,)- 


(6.104) 


Problems 


Problem 6.1 Root Finding Methods 

This computer experiment searches roots of several test functions: 
f{x) = x n - 2 n = 1, 2, 3, 4 (Fig. 6.10) 
fix) = 5 sin(5x) 
fix) = (cos(2x)) 2 — x 2 
fix ) = 5{VW+2\ - 1) 
fix) = e~ x ln.r 
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f{x) = (x- l) 3 (Fig. 6.11) 
fix) = x 25 (Fig. 6.12) 

You can vary the initial interval or starting value and compare the behavior of 
different methods: 

• bisection 

• regula falsi 

• Dekker’s method 

• Brent’s method 

• Chandrupatla’s method 

• Newton-Raphson method 

Problem 6.2 Stationary Points 

This computer experiment searches a local minimum of the Rosenbrock function' 1 


h(x, y ) = 100(y - x 2 ) 2 + (1 - x) 1 . (6.105) 

• The method of steepest descent minimizes h(x, y) along the search direction 

s[ n) = -g ( x n) = -400 x(x 2 - y n ) - 2(x n - 1) (6.106) 

4 '° = ~9y n) = -200 (y n - x 2 ). (6.107) 

• Conjugate gradients make use of the search direction 

4 n) = -9? + Pns?-" (6.108) 

4" ) = -4" ) +/3„4"“ 1) . (6.109) 


• The Newton-Raphson method needs the inverse Flessian 


H - 1 


1 

det(//) 


( hyy h X y 

—h xy h xx 


det (H) = h xx h yy - h\ y 

h xx — 1200.V 2 — 400y + 2 h yy — 200 h xv 


—400.v 


and iterates according to 

(;:::M;iK 1 (I) 


( 6 . 110 ) 

( 6 . 111 ) 

( 6 . 112 ) 


(6.113) 


You can choose an initial point (xq, yo). The iteration stops if the gradient norm 
falls below 10~ 14 or if the line search fails to find a lower function value. 


5 A well known test function for minimization algorithms. 




Chapter 7 

Fourier Transformation 


Fourier transformation is a very important tool for signal analysis but also helpful 
to simplify the solution of differential equations or the calculation of convolution 
integrals. In this chapter we discuss the discrete Fourier transformation as a numer¬ 
ical approximation to the continuous Fourier integral. It can be realized efficiently 
by Goertzel’s algorithm or the family of fast Fourier transformation methods. Com¬ 
puter experiments demonstrate trigonometric interpolation and nonlinear filtering 
as applications. 


7.1 Fourier Integral and Fourier Series 

We use the symmetric definition of the Fourier transformation: 



(7.1) 


The inverse Fourier transformation 



(7.2) 


decomposes /(f) into a superposition of oscillations. The Fourier transform of a 
convolution integral 



(7.3) 


becomes a product of Fourier transforms: 



/ 


'OO 


/ 


‘OO 


dt' f(t')e- iut ’ 


hit - f')e _iw(f_, ) £/(f - f) 


—OO 


—OO 
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= V2^f(u)h(uj). (7.4) 

A periodic function with f(t + T) = f(t) 1 is transformed into a Fourier series 

f(t)= ^ e lw "'/(w„) with u>„ = n —, /(w„) = - / /(f)e _laV </f. (7.5) 


For a periodic function which in addition is real valued /(f) = /(f)* and even 
/(f) = /(—f), the Fourier series becomes a cosine series 


/(f) = /(wo) + 2 ^ /(w„) cos w„f 

n =1 

with real valued coefficients 

1 f T 

f(^n) = — I f(t) COS U) n tdt. 

1 Jo 


(7.6) 


(7.7) 


7.2 Discrete Fourier Transformation 


We divide the time interval 0 < f < T by introducing a grid of N equidistant points 


t„ = nAt = n — with n — 0, 1, ... IV — 1. 
N 


The function values (samples) 

fn = f(t n ) 

are arranged as components of a vector 

/ /<o \ 

f = : 

With respect to the orthonormal basis 

So ,n \ 


e„ = 


, n = 0, 1,... N - 1 


(7.8) 


(7.9) 


(7.10) 


s Sn-i,„ J 


*This could also be the periodic continuation of a function which is only defined for 0 < t < T. 
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f is expressed as a linear combination 

iv -1 

f=X/" e «- (7 ' U) 

n= 0 

The discrete Fourier transformation is the transformation to an orthogonal base in 
frequency space 



N -1 



( 7 . 12 ) 


n =0 




with 


27T 


( 7 . 13 ) 



These vectors are orthogonal 


N -1 


l e iO-/)27T 



l_e i O-y / )27T/A( 


^ =0 /Of 7-/^0 
IV for j-j' = 0 


( 7 . 14 ) 



( 7 . 15 ) 


Alternatively a real valued basis can be defined: 





jmax = f (even N) y max = ^(odd N). 


'max 


( 7 . 16 ) 


The components of f in frequency space are given by the scalar product 


N -1 N -1 N -1 



( 7 . 17 ) 


From 


iV-l 



( 7 . 18 ) 
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we find the inverse transformation 


fn 


1 

N 


N -1 


Z 

7=0 


1 

N 


N -1 

Z £. 

7=0 




(7.19) 


7.2./ Trigonometric Interpolation 


The last equation can be interpreted as an interpolation of the function /(f) at the 
sampling points t„ by a linear combination of trigonometric functions 


N -1 


/«> = £ X A, 


(7.20) 


7=0 

which is a polynomial of 


1 — / 

q = e t\ 


(7.21) 


Since 

0 — = e -if jn = e i 2 f(N-j)n = ( 7 .22) 

the frequencies u/ y - and are equivalent (Fig. 7.1) 


JV-l 


V-l 


/a,,-; = Z fn^ N ~ i)n - Z = f"-r 


n =0 


n =0 


(7.23) 


Fig. 7.1 (Equivalence of wi 
and i) The two functions 
coswf and cos(M — ljojf 
have the same values at the 
sample points f„ but are very 
different in between 
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time t/T 



frequency index j 

Fig. 7.2 (Trigonometric interpolation) For trigonometric interpolation the high frequencies have 
to be replaced by the corresponding negative frequencies to provide meaningful results between the 
sampling points. The circles show sampling points which are fitted using only positive frequencies 
(full curve ) or replacing the unphysical high frequency by its negative counterpart (broken curve). 
The squares show the calculated Fourier spectrum. See also Problem7.1 


If we use trigonometric interpolation to approximate /(f) between the grid points, 
the two frequencies are no longer equivalent and we have to restrict the frequency 
range to avoid unphysical high frequency components (Fig. 7.2): 


2ir N -1 < , < 

" ~T T~ - U J - 


27T N -1 
T 2 


N odd 


2tt N ^ ^ 2-k a tv 

— ~Y ~2 — — ~T * “ 


(?-■) 


N even. 

The interpolating function (N even) is 


fit) 

fit) 


£L -1 

T 1 


j=-K 
J 2 

N -1 

V Z 


even N 


odd N. 


(7.24) 


(7.25) 


(7.26) 


The maximum frequency is 
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_ 2tt TV 

w mai = — 

and hence 


fm 


1 _ N 
2^ Wmax - 2 T 


U 
2 ' 


(7.27) 


(7.28) 


This is known as the sampling theorem which states that the sampling frequency f s 
must be larger than twice the maximum frequency present in the signal. 


7.2.2 Real Valued Functions 


For a real valued function 


fn = /; 

and hence 


f 


( N -1 \* N -1 

Z fn*-"*) =Z fn^=7^ 

n =0 / «=0 


(7.29) 


(7.30) 


Here it is sufficient to calculate the sums for j = 0,... /V/2. If the function is real 
valued and also even 


f-n = fn 

N-l N—l 

Lj - Z = Z 


12=0 


12=0 


(7.31) 

(7.32) 


and the Fourier sum (7.19) turns into a cosine sum 

fn = 


2 M 


1 


1 2 Z-: ~ (It: \ 

- fun H- s fu cos I - in ) odd 

- 1' 0 2M - 1 ^ J 1 \2M -V ) 


fn 2 M^° ' M 


' ~ 7T 1 

Z L. cos ( — jn ) H-cos(H7r) even TV = 

J \M ) 2 M M 

7=i 


TV = 2M - 1 

(7.33) 
= 2 M 

(7.34) 


which correspond to two out of eight different versions [70] of the discrete cosine 
transformation [71, 72]. 
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Equation 7.34 can be used to define the interpolating function 



The real valued Fourier coefficients are given by 


(7.35) 


M— 1 

Kti = fo + 2 y'. fn cos (ujjtn) odd N = 2M — 1 (7.36) 

n= 1 
M— 1 

fw, = /o + 2 y /„ cos (ijJjtn) + fu cos(y'Tr) even IV = 2M. (7.37) 

« = 1 


7.2.3 Approximate Continuous Fourier Transformation 

We continue the function /(f) periodically by setting 

/v = /o (7.38) 

and write 


'V - 1 | | 

f UJ = Z = o /o + + ' " e-^^/v-i + T/at- 

n=0 

Comparing with the trapezoidal rule (4.13) for the integral 

J o e-^fWdt « ( ~e _i “ j0 /(0) + 

+ ■ ■ • + - 1)) + \f(T)\ 

we find 


(7.39) 


(7.40) 


/>,) = ^ ^ e-^mdt « (7.41) 

t Jo tv 

which shows that the discrete Fourier transformation is an approximation to the 
Fourier series of a periodic function with period T which coincides with /(f) in the 
interval 0 < t < T. The range of the integral can be formally extended to ±oo by 
introducing a windowing function 
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W(t) = 


for 0 < t < T 
0 else 


(7.42) 


The discrete Fourier transformation approximates the continuous Fourier transforma¬ 
tion but windowing leads to a broadening of the spectrum (see p. 145). For practical 
purposes smoother windowing functions are used like a triangular window or one of 
the following [73]: 

i / „-( W _i )/2 \ 2 

W(t„) — e 2 v "(v—i )/ 2 ) a < o.5 Gaussian window 
W(f„) = 0.53836 — 0.46164 cos Flamming window 

W(t n ) — 0.5 (l — cos (^rj-)) Flann window. 


7.3 Fourier Transform Algorithms 

Straight Forward evaluation of the sum 



f„ + isin 



fn 


needs 0(N 2 ) additions, multiplications and trigonometric functions. 


(7.43) 


7.3.1 Goertzel’s Algorithm 

Goertzel’s method [74] is very useful if not the whole Fourier spectrum is needed 
but only some of the Fourier components, for instance to demodulate a frequency 
shift key signal or the dial tones which are used in telephony. 

The Fourier transform can be written as 

N - 1 . / / 

Z „ : 2tt „ 27T1 ; / „ 27ti ; / 2m ; „ 

/«e »' =/o + e "^l/i+e » 1 f 2 ■ ■ ■ ( /jv -2 + e » J f N -i 

n =0 ' ' 

(7.44) 



which can be evaluated recursively 


TiV-l — fN-\ 

y n = fn + er^ i y n +\ n = N- 2, ...0 


(7.45) 
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to give the result 

Lj = To- (7.46) 

Equation (7.45) is a simple discrete filter function. Its transmission function is 
obtained by application of the z-transform [75] 

OO 

u(z) = Y J u nZ~ n (7.47) 

11 =0 


(the discrete version of the Laplace transform) which yields 

Hz) 


y{z) = 


i —— i 

1 — ze n j 


(7.48) 


One disadvantage of this method is that it uses complex numbers. This can be avoided 
by the following more complicated recursion 


U N +1 — U N — 0 


u„ — f„ + 2n„ + i cos ^k — u n+ 2 for n = N — 1,... 0 

with the transmission function 

u(z ) _ 1 _ 

Tfe) l-j/jf'+c'T'l+ji 


A second filter removes one factor in the denominator 


y(z) 

u(z) 


= ^1 — ze * 


which in the time domain corresponds to the simple expression 


y„ = u n - e « J u n+ 1 . 


The overall filter function finally again is (7.48). 

y(z) _ 1 

f(z) ~ 1 -ze-T^' 


(7.49) 


(7.50) 


(7.51) 


(7.52) 
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Hence the Fourier component of f is given by 

a 27ri • 

fuj = yo = uo-e~ trJ ui. (7.53) 

The order of the iteration (7.44) can be reversed by writing 

= (7-54) 

which is very useful for real time filter applications. 


7.3.2 Fast Fourier Transformation 

If the number of samples is N — 2 P , the Fourier transformation can be performed 
very efficiently by this method. 2 The phase factor 

e~if y'm = W J N m (7.55) 

can take only N different values. The number of trigonometric functions can be 
reduced by reordering the sum. Starting from a sum with N samples 


JV—1 


F N (f 0 ...f N . l ) = Y J fnK n 

n =0 


(7.56) 


we separate even and odd powers of the unit root 

K — \ — — 1 

F N (f 0 . . . f N - 1 ) = Z f 2 ,nW J N 2 ” 1 + Z f 2 m+lW J N (2m+1) 

m =0 m=0 


f-i f-i 

= Z h,n^ im + K Z hm + xe~^ jm 

m= 0 m= 0 

= F N/2 (fo, f 2 - ■ ■ fN—i) + W J N F N / 2 (fi, h ■ ■ ■ In- l)- 

This division is repeated until only sums with one summand remain 

Fi(f„) = f„. 


(7.57) 


(7.58) 


2 There exist several Fast Fourier Transformation algorithms [76, 77], We consider only the simplest 
one here [78]. 
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For example, consider the case N = 8: 

Wo ■ • • fi) = Wo/2/4/6) + W1/3/5/7) 

Wo/2/4/6) = W0/4) + W W2/6) 
W1/3/5/7) = W1/5) + W3/7) 

W0/4) = /o + 1^/4 
W2/5) = /2 + wife 
Fiififi) = fi + wife 
Fiififi) = fi + Wifi. 


Expansion gives 


= /o + w/ fi + wi / 2 + w 4 7 ' wi fi 


+ wi fi + u;' wi fi + wi wl fi + wi wi wi fi. 


(7.59) 


Generally a summand of the Fourier sum can be written using the binary 
representation of n 

n = ^li U = 1,2,4, 8... (7.60) 


in the following way: 

fie-^F = f n e~^(h+h+-)j = f n W J N/h W J N/h .... ( 7 . 61 ) 

The function values are reordered according to the following algorithm 

(i) count from 0 to N-l using binary numbers m = 000 , 001 , 010 ,... 

(ii) bit reversal gives the binary numbers n = 000, 100, 010,... 

(iii) store /„ at the position m. This will be denoted as s m = fi 
As an example for N=8 the function values are in the order 


M 

51 

52 

53 

5 4 

55 

56 

V S7 ) 


//o\ 

fi 

h 

fi 

h 

fi 

fi 

\fi J 


(7.62) 


Now calculate sums with two summands. Since Wj can take only two different values 
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w{ 


1 for j = 0, 2, 4, 6 
-1 for j = 1,3, 5, 7 


(7.63) 


a total of 8 sums have to be calculated which can be stored again in the same 
workspace: 


//o + M 


/ s 0 + W%s\ \ 

/o - /4 


So + W^Sl 

/2 + /6 


S2 + W 2 2 s 3 

fi - u 


Si + W 2 3 s 3 

fi + fs 


s 4 + w}s 5 

fl-f5 


s 4 + W 2 5 ^5 

h + fi 


S 6 + w}s 7 

\ h ~ fi ) 


^6 + Wjsi ) 


Next calculate sums with four summands. W 4 can take one of four values 


1 for 7=0,4 
j _ -1 for 7=2,6 

4 W 4 for 7 = 1,5 ' 
-Vk 4 /or 7 = 3,7 

The following combinations are needed: 


/ /0 + /4 + C/2 + /6 ) \ 


( so + W 4 s 2 \ 

/o + /4 — (/2 + A) 


si + w 4 7 3 

(7o - / 4 ) + W 4 (/ 2 - f 6 ) 


5 0 + W|s 2 

(7o - / 4 ) - W 4 (f 2 - f 6 ) 


Si + W 4 s 3 

fi + fi + (/ 3 + fn) 


S 4 + W 4 S 6 

fi + fs — (h + fj) 


S 5 + W 4 S 7 

(/t - fs) ± W 4 (/ 3 - / 7 ) 


s 4 + W 4 6 s 6 

V(/ 1 -/5)±W 4 (/ 3 -/7V 


VS5 + Wjs 7 j 


(7.65) 


(7.66) 


The next step gives the sums with eight summands. With 


W{ 


1 7 = 0 
W 8 7 = 1 
7=2 
^8 7=3 
-1 7=4 
-W 8 7 = 5 
-^ 8 2 7 =6 
~Wi 7=7 


(7.67) 
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we calculate 


/ /o + h + (/2 + h) + (/l + /5 + (/3 + m \ 


(s 0 + W°s 4 \ 

fo + h - (72 + h) + w 8 (/j + fs - (/3 + m 


*1 + Wgis 

(/o - f\) + W 4 (/ 2 - f 6 ) + M/ 8 2 (/! - f 5 ) ± W 4 (f 3 - fj) 


J2 + 

(.fo - U) - W 4 (f 2 - /6) + W 8 3 ((/i - / 5 ) ± W 4 (/ 3 - /?)) 


*3 + W g 3 J 7 

fo + h + (/2 + /6) - (/l + fs + (/3 + /?)) 


jo + Wg j 4 

fo + U - (/2 + ./is) - W 8 (/i + / 5 - (/ 3 + /?)) 


Jt + j 5 

(/0 - /4) + W 4 (/ 2 - /6) - W|((/i - / 5 ) ± W 4 (f 3 - / 7 )) 


J 2 + M/|j 6 

\ (To - /4) - W 4 (/2 - 76) - w|((/i - / 5 ) ± W 4 (/ 3 - / 7 )) J 


+ Vk 8 j 7 / 


(7.68) 


which is the final result. 

The following shows a simple Fast Fourier Transformation algorithm. The number 
of trigonometric function evaluations can be reduced but this reduces the readability. 
At the beginning Dcita[k] are the input data in bit reversed order. 

size:=2 
first :=0 

While first < Number_of_Samples do begin 
for n:=0 to size/2-1 do begin 
j:=first+n 
k:=j+size/2-l 

T:=exp(-2*Pi*i*n/Number_of_Samples)*Data[k] 

Data[j]:=Data[j]+T 
Data [ k ]: =D ata [k] -T 
end; 

first:=first*2 

size:=size*2 

end; 


Problems 

Problem 7.1 Discrete Fourier Transformation 

In this computer experiment for a given set of input samples 

f n = fin 1 -) n = 0... N — 1 (7.69) 

• the Fourier coefficients 


N -1 


2n 


f» i = '£f« e iUi ‘" UJ J = ^rJ 


n =0 


j = o ... N — 1 


(7.70) 
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are calculated with Gortzel’s method 7.3.1. 
• The results from the inverse transformation 


N -1 


- N X! 


= N n J 


j =0 


(7.71) 


are compared with the original function values 
• The Fourier sum is used for trigonometric interpolation with only positive 
frequencies 


m = 


i _ 

N X A 


7=0 



(7.72) 


• Finally the unphysical high frequencies are replaced by negative frequencies 
(7.24). The results can be studied for several kinds of input data. 



signal 


rectangle 


=c>= 


- 


cutoff 

=P= 


Fig. 7.3 (Screenshot from computer exercise 7.2) Top The input signal is rectangular with Gaussian 
noise. Middle The Components of the Fourier spectrum ( red ) below the threshold (green line ) are 
dropped. Bottom the filtered signal is reconstructed (blue) 
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Problem 7.2 Noise Filter 

This computer experiment demonstrates a nonlinear filter. 

First a noisy input signal is generated. 

The signal can be chosen as 

• monochromatic sin(wt) 

• the sum of two monochromatic signals a\ sin u>\t + a 2 sino^f 

• a rectangular signal with many harmonic frequencies signtsin uit) 

Different kinds of white noise can be added 

• dichotomous ± 1 

• constant probability density in the range [—1, 1] 

• Gaussian probability density 

The amplitudes of signal and noise can be varied. All Fourier components are 
removed which are below a threshold value and the filtered signal is calculated 
by inverse Fourier transformation. Figure 7.3 shows a screenshot from the program. 


Chapter 8 

Time-Frequency Analysis 


Fourier-analysis provides a description of a given data set in terms of monochromatic 
oscillations without any time information. It is thus mostly useful for stationary sig¬ 
nals. If the spectrum changes in time it is desirable to obtain information about the 
time at which certain frequencies appear. This can be achieved by applying Fourier 
analysis to a short slice of the data (short time Fourier analysis) which is shifted along 
the time axis. The frequency resolution is the same for all frequencies and therefore it 
can be difficult to find a compromise between time and frequency resolution. Wavelet 
analysis uses a frequency dependent window and keeps the relative frequency reso¬ 
lution constant. This is achieved by scaling and shifting a prototype wavelet - the so 
called mother wavelet. Depending on the application wavelets can be more general 
and need not be sinusoidal or even continuous functions. Multiresolution analysis 
provides orthonormal wavelet bases which simplify the wavelet analysis. The fast 
wavelet transform connects a set of sampled data with its wavelet coefficients and is 
very useful for processing audio and image data. 


8.1 Short Time Fourier Transform (STFT) 


Fourier analysis transforms a function in the time domain fit) into its spectrum 



( 8 . 1 ) 


thereby losing all time information. Short time Fourier analysis applies a windowing 
function 1 (p. 133) e.g. a simple rectangle (Fig. 8. 1) 2 


'Also known as apodization function or tapering function. 

2 There are two different definitions of the sine function in the literature. 
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1 

0.8 

0.6 g 

0.4 1 ^ 

C4 

°-2 "l 
0 


- 0.2 


Fig. 8.1 (Rectangular window) The rectangular (uniform) window and its Fourier transform are 
shown for d = 1 



Fig. 8.2 (Triangular window) The triangular (Bartlett) window and its Fourier transform are shown 
ford = 1 


w R (t) = 


Wr(u) = 


1 for \t\ < d 
0 else 
2d sin uod 
~J2tt ud 


2d 


\[2/k 


sinc(wt/) 


or triangle (Fig. 8.2) 


W Tr (t) = 


W Tr (co) = 


(l-^)for t<d 
0 else 

d 2(1 — cos cod) 
V27T OJ 2 d 2 



( 8 . 2 ) 

(8.3) 


(8.4) 

(8.5) 
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3 

$ 


(N 



Fig. 8.3 (Gaussian window) The Gaussian window and its Fourier transform are shown for d = 1 


A smoother window is the Gaussian (p. 192) (Fig. 8.3) 

1 ' 2 


W G (t) = 

with 

W g (oj) = 


dV2n 


exp 


y/2ir 


exp 


r 

'id 2 


u 2 d 2 


For the Gaussian window the standard deviations ’ 
1 

o t — d — ~ 
d 

obey the uncertainty relation 4 


( 8 . 6 ) 


(8.7) 


( 8 . 8 ) 


(8.9) 


Since the Gaussian extends to infinity, it has to be cut off for practical calculations. 
Quite popular are the Hann(ing) and Hamming windows (Fig. 8.4) 


, / 7 Tt\ /I 1 TTt\ 

Wh u „„ (f) = cos ( S j = (2 + 2 COS 7) 


( 8 . 10 ) 


3 FIere we use the definition <r 2 = dt W(t)t 2 . If instead a 2 = dt \W(t)\ 2 t 2 is used then 
a, = ^and ^ 

4 For a Gaussian the time-bandwidth product is minimal. 
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Fig. 8.4 (Hann and Hamming window) The Hann (full curves) and Hamming (dashed curves) 
windows together with their Fourier transforms are shown for At = 1. The Hamming window is 
optimized to reduce the side lobes in the spectrum 


w Hann(^) = 


d sincwc/ 



( 8 . 11 ) 


W Hamm 10 — 



23 7 Tt 

— cos- 


54 d 


) 


( 8 . 12 ) 


, 1 4 u 2 d 2 

~ CL J- ryn 2 

W H amm(u) = —= -sincwc/. 

V 27T 1-^2 

For a general real valued function 


(8.13) 


W(uj)* = 


a 


\[2tt J -o 


W(t)e~ lu], dt) = 


V - 1 

/ 


\[2 tt 7-o 


W(t)e luJ, dt = W(-oj) 


(8.14) 


and for an even function 
1 


W(uj) = 


s/7/k . 


W(t)e~ lw dt = 


s/2n . 


W(tW w, d(-t) = W(-uj). 


—oo 


(8.15) 
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If W (f) is real and even than this holds also for its Fourier transform 

~ * 1 

W(—uj) = W{oj) = W{u) = / W(t)cos(ut)dt. ( 8 . 16 ) 

V 27 T 7 —oo 

The short time Fourier Transform 

1 r°° 

X(t 0 , tu) = T[W*(t - to)f(!)](w) = - 7 = / W*(t - t 0 )f(t)e~ lwt dt ( 8 . 17 ) 

V 27 T 7 —oo 

depends on two variables to and u>. Since it has the form of a convolution integral it 
becomes a product in Fourier space, where 


1 r . i r°° i r°° 

/ r/toe- 1Wo ' o Z(t 0 , u>) = -= / £/t 0 e- Iwof ° dtW*(t- f 0 )/(f)e 

2-7T 7 V27T 7-oo ■sj'2/K 7-00 


y/27T 

r r 

7—oo 7- 


<*rf(f - t 0 )e iwo<f_fo) W*(t - f 0 )/(f)e“ 


■Iff 

2 tr V_ c 


rl(r-/o)e“ iwo( '“' o) W(t-to) 


/ OO \ * 

-oo/ 


r/t /(f)e 


-iwor — 


= W*(oj 0 )/(a; + u;o). 


(8.18) 


For a real valued an even windowing function like the Gaussian ( 8 . 6 ) the STFT can 
therefore be calculated from 


X(to,u>) = 


1 


\/2n . 


dujQ& luJ ° 0 W (uj 0 )f(u) Q + w). 


Alternatively, the STFT can be formulated as 
1 f°° 

X(t 0 , to) = —=e luJ, ° / f(t) W*(t - to)Q- luJ(, - ,o) dt 

V 27 T J—oo 


(8.19) 


( 8 . 20 ) 


1 




\[2t: 


f(t)V(t-to)dt 


which involves a convolution of f(t) with the wave packet (Fig. 8.5) 
£2(t- t 0 ) = W(t - fo)e iw(,_ ' o) 


( 8 . 21 ) 


which is localized around to. In frequency space the wave packet becomes a band 
pass filter 
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Fig. 8.5 (Gabor wave packet) Left Real (full cum) and imaginary (dashed curve) part of the wave 
packet (8.21) are shown for a Gaussian windowing function with oj = 5 and2J 2 = 3. Right In the 
frequency domain the wave packet acts as a bandpass filter at ljq = oj 


f2(uJ 0 ) 


1 


\/27r . 


dte~ iul0 'W(t)e iL 


\Tht J 

= W(ujo — u>) 


cite. 


-i(uo-UJ)f 


W(t) 


J poo poo 

— dt 0 e-'^ I f(t)S2*(t-t 0 )dt 

■‘‘- 7 T ./_on , 1—00 


J poo poo 

- dt d(t — to) f(t)e~ 1Ua ' f2*(t — fo)e iw ° (,_to) 
^ J —oo J — oo 


~2n dt 


( 8 . 22 ) 


X(t 0 ,u) = 

1 f 


piwto 


t/w 0 e IW »' 0 /(wo)^*(a;o) 


\/ 27 T J-, 

i r 


•Jl/rt 

X) 

t/woe i( “ +Uo), °/(wo)^*(wo) 


\/27r 7-o 


dojQe 0 f (ujq - u>)f2 (ujq - w). 
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The STFT can be inverted by 


duoX(to, oj)e i “ f ° = ~^= J dt j 


dtue iuJ '°W*(t -t 0 )f(t)e 


= -4= [ dtW*{t - t 0 )/ (t)2nS(t - t 0 ) = V2fW*(0)f(t 0 ) 

V2tt J 

or alternatively 

J dt 0 j diuX(t 0 ,uj)e iujt = J dt 0 J r/w—L=e iwf J W* it' - f 0 )/(t')e _i “ f ' dt' 


(8.23) 


: f dt 0 J= f°° 2-K5{t-t')W*{t’ -to)f{t')dt' 

J V27T j-oo 

: J dt 0 -J=2nW*(t - to)f(t) = -fhtfit) J W*(t')dt '. 


(8.24) 


STFT with a Gaussian window is also known as Gabor transform [79] which is 
conventionally defined as 


/ OO 

* e -“"('-'o )2 e -i“7 ( 0- 

-OO 


(8.25) 


Example: Spectrogram 


The STFT is often used to analyze audio signals. Let us consider as a simple example 
a monochromatic signal, which is switched on at time t — 0 


fit) = 


0 t < 0 
sin(oj s t) t > 0. 


(8.26) 


Using a Gaussian window, the Fourier transform can be calculated explicitly (an 
algebra program is very helpful) 


I r°° 

X(t 0 , to) = - / dt e -i^e- ( '- fo) P* sin (to s t) 

2nAt J 0 


i 

4a/27t 


e -it 0 (w-w s ) e -d (w-w s ) z /2 


_|_ 1 e -if 0 (M+M,) e -rf 2 (M+M,) 2 /2 


(" f (‘ 

(e,f(i 


iAt 2 (to - to s ) - t 0 


V2d 

i d 2 (to + to s ) — t 0 

V2d 


B 

)-) 


(8.27) 
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Fig. 8.6 (3-d spectrogram) The squared magnitude of the STFT (8.27) is shown for u s = 5 and 
2d = 1 


There are two contributions since the real function /(f) contains oscillations with 
± oj s . The squared magnitude 3 \X(to, to) | 2 is shown as a 3-d spectrogram in Fig. 8.6. 
The width of the window determines the resolution both in time and frequency. 
Neglecting interference terms, at resonance w = u> s the squared magnitude rises 
according to 


(8.28) 

and reaches its stationary value within a time of ta 2d, whereas in the stationary state 
at to y>> cl, the dependency on the frequency mismatch Auj = uj — u> s is given by a 
Gaussian with a width of 2cr w = y/2/d. The dependence of time and frequency res¬ 
olution on the window width is shown qualitatively by 2-dimensional spectrograms 
in Fig. 8.7. 


8.2 Discrete Short Time Fourier Transform 

The continuous STFT is very redundant and not useful for the analysis of data which 
are sampled at discrete times. Therefore we introduce a series of overlapping windows 
centered at equidistant times t n — nAt (Fig. 8.8) 

W n (t) = W(nAt - t). (8.29) 


5 This is a measure of the spectral power distribution. 
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2d 2 =0.25 


2d 2 =0.5 


2d 2 =1.0 


2d 2 =2.0 




Fig. 8.7 (Spectrograms with different window width At) The squared magnitude of the STFT 
(8.27) is shown for uj s = 5 and 2d 2 = 0.25, 0.5, 1.0, 2.0. For larger values of the time window d 
the frequency resolution becomes higher but the time resolution lower 


Assuming that the windowing function W n {t) = 0 outside the interval [t n — 
d, t n +d] we apply (7.5) and expand W n (t)f(t) inside the interval as a Fourier series 


oo 

9n(t) = W n (t)f{t) = X e 1Um ‘g nm with u m = m ^ \t-t n \<d. (8.30) 

m =—oo 

We extend this expression to all times t by introducing the characteristic function 
of the interval 


Xn(t) = 


1 for \t — t n \ < d 
0 else 


OO 

g n (t) = w n {t)f(t) = Xn(f) X! 

m ——oo 


(8.31) 

(8.32) 


The Fourier coefficients, given by the integral 


Fig. 8.8 (Discrete STFT) 
Assuming that the 
windowing function 
W n (f) = 0 outside the 
interval [t n — d,t„ + d] we 
apply (7.5) and expand 
W„(t)f(t) inside the interval 
as a Fourier series 


2d 


W^t) w n (t) W n+1 (t) 
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1 rtn+d 

Qnm — 7T~] / W(t - t n ) f {t)& lu ’ m 'dt (8.33) 

2 d J tn _ d 


obviously correspond to the STFT at times f„and frequencies uj m 


\ ft„+d 

-j=\ W(f — t n ) f dt 

V27T Jt„-d 

(8.34) 


If the windows are dense enough such that there union spans all times, the signal can 
be reconstructed by summation 


1 

X(t B ,u m ) = -= / W(t- t^fOW^'dt = 
V 2.7T J —OO 

2 d „ 

— /- 9nm • 

V27r 


= fit) Y, w„(t) = X XnitW^gnm. (8.35) 

n n nm 

This expression simplifies, if 


^ W n (t) = const (8.36) 


which is e.g. the case for triangular windows as well as the Hann and Hamming 
windows with At — d (Fig. 8.9). 

For practical applications, we assume that the function git) has been sampled at 
N equidistant times within the interval [ t n — d, t n + d] 

2d 

r ns = t„ — d + s — s — 0, 1,... N — 1 (8.37) 

N 



Fig. 8.9 (Window functions with constant sum) For the triangular (Left) and the Hann and Ham¬ 
ming (Right) window the sum y „ W n (t) becomes constant (dotted lines) if the windows are shifted 
by half their width At = d 
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7T Z7T 

Uj = -jj, U j(T n , s - t„ +d) = js — 

and apply the discrete Fourier transformation method (p. 131) 


(8.38) 


N -1 


At—1 


a -Vo t -'^i^-in+d) _y -ijsf 
yn,u)j — x , yn,s^ — / 1 yn,s c 

s=0 s=0 

t N-l t N-lN-l 

— 'Va e ijs % - -Wo ,e,- ijs '%Q ijs % — a 

N yn,uj e — N ^ ^ y n ,s f e e — 9n,i 


(8.39) 


N 


(8.40) 


7=0 5=0 


7=0 

Example: FM Signal 

Figures 8.10 and 8.11 show the STFT analysis of a frequency modulated signal 


fit) = sin 0(7) 


= sin 


au>o i 

ujot H-(1 — COSUq?) 

W1 ) 


(8.41) 


with a momentaneous frequency of 
d<P 

uj(t) = —— — u7o(l + a sinwif) (8.42) 

at 

for carrier frequency ^ = 10 kHz, modulation frequency = 25 Hz(100Hz) and 
modulation depth a = 0.3. 


f [kHz] 



0 50 

t [ms] 


t [ms] 



0 10 20 


f [kHz] 


Fig. 8.10 (STFT analysis of a FM signal) The figure shows screenshots from Problem 8.1. Left 
spectrogram Right STFT spectra. Sampling frequency is 44100FIz, number of samples 512, Hann 
windows are used with a shift of 8 samples (0.18 ms) between neighbor windows. 6 ms time res¬ 
olution and 1.1 kHz frequency resolution are sufficient to resolve the 25 Hz modulation. The time 
dependent spectra have their smallest width at the stationary points of the momentaneous frequency 
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Fig. 8.11 (STFT analysis of a FM signal) The figure shows screenshots from Problem8.1. Left 
spectrogram Right STFT spectra. Sampling frequency is 44100F[z, number of samples 512, Hann 
windows are used with a shift of 2 samples (0.045 ms) between neighbor windows. 6 ms time 
resolution and 1.1 kHz frequency resolution are not sufficient to resolve the 100 Hz modulation. 
Only minimum and maximum frequencies are observed 


8.3 Gabor Expansion 


For the special case of rectangular windows with distance At — 2d b 


W n (t) = W R (t - 2dn) = Xn (t) (8.43) 

y W n (t) = 1 = const (8.44) 


we have 


f(f) = y, g„ ( t ) 


n 

oo 


= X Z Xn(t)^ m ‘ g nm 

n m=—o o 


(8.45) 

(8.46) 


This equation expands /(f) and its Fourier transform as linear combinations of 
elementary “signals” which are located at t„ in time and uj m i n frequency 

h n ,n = x«e iw " f . (8.47) 


6 For simplicity, we do not normalize the window here. 















8.3 Gabor Expansion 


157 


2 d _ \ 

h nm — _ e lwia; 0Jm) sine d(uj — uj m ) 

VZ7T 


f (0 — ^ ' hn,m9n,m f 'O^O — ^ ^ 


■if]tun • 


(8.48) 

(8.49) 


A similar expansion is obtained if we use rectangular windows in Fourier space with 
width and distance Aw [80] 


2Au> 

h nm (t ) = -p= sine ((f - t„)4w)e“" 

V27r 

and sample the spectrum at times 

t n = rc-p- (8.50) 

Auj 


to obtain 


OO 

/(w) = M/mM = X x-(^/„ (8.51) 

n=—o o 


puj m +Auj 

fnm = W— / /(W)^. (8.52) 

Gabor [79] discussed an expansion with Gaussian signals (Fig. 8.5). In general, how¬ 
ever, the elementary signals are not orthogonal which makes the determination of the 
coefficients a„ m complicated. Bastiaans [80, 81] introduced another auxiliary set of 
elementary signals 

7 n, m = 7 (t - «4t)e“-' (8.53) 

which are biorthogonal, i.e. 


J In' f m'(f')hn,m(t')dt — $n,n'^m,m' (8.54) 

and allow the calculation of the Gabor expansion coefficients from a scalar product 
j f ^) dt = X! j a nm'rt, m hnm( t )dt = Cl nm . (8.55) 
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Determination of 7 for a given windowing function can be simplified by application 
of the Zak transform [82], Discrete versions of the Gabor transform [83] are popular 
in signal, speech and image processing. 


8.4 Wavelet Analysis 


The STFT method uses constant frequency and time resolution. Therefore the lowest 
frequency of interest determines the minimum width of the window whereas at 
higher frequencies shorter time windows could be more appropriate to increase time 
resolution while keeping the relative uncertainty in frequency constant (Fig. 8.12). 
This is the basic idea of the wavelet transform. Whereas STFT uses wave packets of 
the form (8.21) 

ff/b AO = W(t - f 0 )e iw{ '- ,o) (8.56) 





time time time 

Fig. 8.12 (Morlet wavelets and STFT wave packets) Top STFT uses the same window for all 
frequencies Bottom wavelets use a variable window width to keep the form of the wave packet and 
the relative frequency resolution constant (only the real part is shown) 
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where only the oscillating part is scaled with frequency to, wavelets scale the whole 
function like in 


!2 la , s (t) = W 


(R) 


,iw 0 (r-fo)/s 


or, more generally 


n, 0 At) — 


f2(w) 


I 


1 


ip 


(R) 


(8.57) 


(8.58) 


\[2tt J-. 


OO 


Vkl \ s J 


dt 


\/27T J-, 


oosign s 

e -Mst’+to)__^^ d ^ st ' + fo) 

oosigns 

poo 




1 

= VRe- iw '° -j= \ e-“'V(tVl' = VW- iw '°£(sw) (8.59) 

V 27T J —oo 

where a whole family of wavelets is derived from the “mother wavelet” P (t) by 
shifting and rescaling. The prefactor has been introduced to keep the norm invariant 

j\n l0 , s (t)\ 2 dt = 1 J\p 1 2 dt 

Y roo signs /» 

= n / \P(t')\ 2 d{st' + t 0 ) = / \*{t')\ 2 dt'. 

|S| J —oosign.? J 


(8.60) 


Closely related to the short time Fourier analysis is the Morlet (or Gabor) wavelet, 
which is also very useful in quantum physics [84], It is defined as 7 


P(t) = W G (t)e' :jJ<, ‘ = 


1 


r l ! 4 Vd 


exp 


r 

"2R 


1 


Vd 

ip( u ) = W c (uj- ujq ) = —r-T exp 

7T 1 ' 4 


-y(w- ujq)- 


(8.61) 


(8.62) 


The similarity of a signal /(f) to a wavelet with scale .v centered at to is measured 
by the correlation integral 


C(t 0 ,s)= / f(tW* os (t)dt = 


1 


VR. 


p 


(R) 


f(t)dt 


(8.63) 


7 The conventional normalization is J dt\P(t)\ 2 = 1. 
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which becomes a product after Fourier transformation with respect to to 


C(ui,s)= —L= [ C(t 0 ,s)e lul, °dt 0 

= -j= J dt 0 e~ iuJ, ° J dt^- J du>'- s /\f\&*(sw')e- iw ' ( - , - ,o) J dui" 

= *J2tt\s\ dt J du/tfi* (,sa/)e -lw ' J duj"f(uj")e lul '5{u — u/)<5(u/ — u") 

= y/2w\s\ I duj’^isu')^'' /(a/)e ia/t 
= y27r|i|<f , *(jw)/(w). 


(8.64) 


For the Morlet wavelet this becomes 


C(u>, s ) = 7 T l/ *j2\s\d exp | — — (sut — wq) 2 


/(w) 


= 7T 1/4 j2\s\d 


exp 


(sdy 




(8.65) 


i.e. C(uj, s) averages the spectrum / over a range with a width of <r w = 1 /sd around 
u> = loq/s and a constant ratio 


<?u> _ 1 

ui ujod 


( 8 . 66 ) 


8.5 Wavelet Synthesis 

For data processing it is necessary to reconstruct the data from the wavelet coefficients 
C (tos). This can be achieved with the help of the integral 8 


r°° r°° l r l l /t-t 0 \ 

/ / -jCffo, s)S2 to ,s(t)dtods = / -jdtods——^ (—-—JC(fo,-s) 

.7—00 J —oo S J S VPl V ^ / 

= -j= j ^ds J dto J rfu)e“ (, -^^(w) J du’e^' 10 v*(suj)f (to) 

= — X = j \ds J dt 0 J sdu"^" { - , - ,a) 'P{uj"s) J duj'e^' ,0 W*(suj')f(uj') 

4 />/‘ 


du"e iuJ " , <I'(u>"s) J dJV*{sJ)f(u')2-K5(u' -to") 


V2n J -ds J sdu/"e ,ul l 'l'(u>"s)'I'(su>")f(uj") 

: J -ds J du;"e iul "'<i'(ij"s)<i'*(suj")f(uj"). 


(8.67) 


'A more rigorous treatment introducing the concept of frames in Hilbert space can be found in [85], 
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If the admissibility condition is fulfilled, which states that the integral 

r °° |</(u;)| 2 


C# = / - duj < oo 

J—oo ^ 

exists and is final, then 


( 8 . 68 ) 


r°° i _ r°° l 

! -ds^(us)^*(ws) = / — = C# 

I—oo S J—oo tU 


and we obtain 


m = 


i 


27rCi 


& J —OO J —OO S 


— C(to, s)£2, 0 ' S (t)dt 0 ds. 


The admissibility condition implies that W (0) = Oandthus j dtd> (t) = 0.Hence, 
the Morlet wavelet (8.61) has to be modified 9 


#(t) = 


V 4 Vd 


Nd exp 


7T 


r 

"M 2 


e‘“ 0 ' — exp 


Vd 

^ 


exp 


-- ^o) 


exp 


N d = 


^1 + exp {— uj 2 d 2 } — 2 exp 


3 2 j2 
—uj d 

)1 

4 

/ J 


07gt/ 2 
2~ 

d 2 n o 
— — (W“ + U! 0 ) 

1 - 1/2 


(8.69) 

(8.70) 

(8.71) 


Another popular (continuous) wavelet is the “Mexican hat” (also known as Ricker 
wavelet or Marr wavelet) which is given by the normalized negative second derivative 
of a Gaussian (Fig. 8.13) 


#(t) = 


r>/ 4 VM 


K) 


~ 2 n/d ry 

^(u>) = -— u; exp 


exp 

u 2 d 2 


f 

"M 2 


(8.72) 

(8.73) 


Example: Wavelet Analysis of a Nonstationary Signal 

The following example shows screen shots from Problem 8.2. The signal consists of 
two sweeps with linearly increasing frequency of the form 


/ 1 . 2 (f) = sin |wi , 2 t + -^r f2 ] 


(8.74) 


9 This correction is often neglected, if the width is large. 
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1 

0.8 
0.6 
04 
9 ^ 0.2 
0 

- 0.2 
-0.4 

-4 -2 0 2 4 

t/d 


0.8 

0.6 



0.2 


-4 -2 0 2 4 

ffld 


Fig. 8.13 (Mexican hat wavelet) Left The mexican hat wavelet is essentially the second derivative 
of a Gaussian. Right Its Fourier transform is a band pass filter around ui max = ±2 /d 




and another component which switches between a 5 kHz oscillation and the sum of 
a 300 Hz and a 20 kHz oscillation at a rate of 20 Hz 


hit) = 


sin(w 2 otHH) + sin(w 30 o H -J) if sin(w 2 o h-J) < 0 
sin(w 5j tH.f) else. 


(8.75) 


The signal is sampled with a rate of 44 kHz and analyzed with Morlet wavelets 
over 6 octaves (Fig. 8.14). The parameter d of the mother wavelet (8.61) determines 
frequency and time resolution. The frequency ujq of the mother wavelet is taken as 
the Nyquist frequency which is half the sampling rate. The convolution with the 
daughter wavelets 


hn.n it) — 



(8.76) 


is calculated at 400 times with a step size of 0.726 ms (corresponding to 32 samples) 
t„=to + nAt (8.77) 


and for 300 different values of the scaling parameter 

s m = 1.015™. (8.78) 

In a logarithmic plot, the relative frequency uncertainty has the same size for all 
stationary signals. 
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Fig. 8.14 (Wavelet analysis) 
Top for d = 1 ms the 
frequency resolution is high 
for the stationary parts of the 
signal. Time resolution is 
low. Middle for d = 0.25 ms 
the pulsating component at 
300 Hz can be resolved but 
time resolution is still poor. 
Bottom For d = 0.0625 ms 
time resolution is sufficient 
to show all the modulations 
while frequency resolution is 
rather poor 
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8.6 Discrete Wavelet Transform and Multiresolution 
Analysis 

The continuous wavelet transform is very redundant and time consuming. Multires¬ 
olution analysis provides a way to define a discrete set of orthogonal wavelets, for 
which the wavelet transform can be calculated very efficiently from a scalar product. 
A discrete wavelet transform uses discrete values of shift and scaling parameters 

5 = a ~ m t 0 = na~ m b (8.79) 

to define the daughter wavelets 10 

• K, n (t) = a m/2 ^ (a m t - nb ). (8.80) 

For integer a , in most cases a = 2, this equation defines wavelets of a multireso¬ 
lution analysis (Fig. 8.15) where m corresponds to the resolution 2 m . 


8.6.1 Scaling Function and Multiresolution Approximation 

At the basic resolution 2° the function f(t) is approximated as a linear combination 

fit) * f(t) (0) = J^fo.n&oAt) (8-81) 


of a scaling function and its translations. 


Fig. 8.15 (Multiresolution 
analysis) Data are analyzed 
with decreasing time 
window At = b/2 1 " 



10 Equation 8.76, in contrast, describes the continuous wavelet transform, which has to be discretized 
for numerical calculations. 
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— <P(t — nb) , n = 0, ±1... (8.82) 

which is chosen [86, 87] such, that the <£o,»f° rm an orthonormal basis of the space 
of linear combinations 

Vo = span{0o,n< n = 0, ±1,...} (8.83) 

J <Pl n (t)4> 0 ,At)dt = 6„, n '. (8.84) 

The best approximation is found by minimizing the norm 

11/(0 - X /0,n^0n(OH 2 = [ (/*(0 - X /(>%<*& (0)(/(0 - Z f0,n'^,At))dt 

n n n ' 

= f |/(0| 2 4t-Z/o»' f /*(0^o,'(04f-Z/on(0 f <?ft,(O/(O* + ^|/ 0 „| 2 (8.85) 

J n' J n J n 

hence by choosing 

fon = J <P* 0n (t)f(t)dt (8.86) 

i.e., the orthogonal projection of f (t) onto Vo. Approximation at the higher resolution 
2'" similarly is given by linear combination 

fit) « = Y, fn.n&rn.nit) (8.87) 

n 

of the scaled functions 

<p mn =2 m/2 0(2 m t-nb) (8.88) 

which form an orthonormal basis for the space 

V m = span {<P m , n ,n = 0, ±1,...} (8.89) 

since 

j K,nit)^n,n'it)dt = 2 m J <P*(2 m t - nb)& (2 m t - n'b)dt 
r clt' 

= 2 m / <P*(t' - nb)0(t' - n’b)— = (8.90) 

The sequence of spaces V,„ is called a multiresolution approximation to the space of 
square integrable functions Z. 2 (R), if [86] 
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(i) ••• C V_i C Vo C Vi C V 2 ... 

oo 

(ii) V,„ is dense in L 2 (K) 

m =—oo 
oo 

(iii) fl V m = {0}. 


(8.91) 

(8.92) 

(8.93) 


Property (ii) has as a consequence, that the approximations f {m) (t) converge to f(t) 
for large m. Hence, due to orthonormality 


/ OO 

^ f(t) 

-OO 


(8.94) 


and the projection operator onto V m 

Pm = X = 2'" (2 m t - nb)0 *(2'Y - nb) -+ 1 (8.95) 


converges to the unit operator. Now, with a > 0 choose the function 
, , . 1 if — a < x < a 

/.<0=| 0dse (8.96) 

Then, 

(■ P,nfab)(t) = X W" 1 - nb ) / 2"'dt'0*(2'"t' - nb) 

n J ~ a 

/ 2 m a 

0*{t' - nb)dt'. (8.97) 

-2"'a 

For large a, the integrals become more and more independent on n, and 

0*{t')dt^ ^0(2 m t-nb) -* 1. (8.98) 

Now we integrate the sum over one period 0 < t < 2 ~ m b and find 


r 2~ m b 


r*(l —n)b 


^0(2 m t - nb)dt = ^ / <P(t) 2 ~ m dt = 2 


0(t)dt 


-nb 


(8.99) 
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and therefore 


uyyuyy 


= b 


( 8 . 100 ) 




— -Jb 


as well as 


I* (0)| = 




( 8 . 101 ) 


( 8 . 102 ) 


a/27T 

Fourier transformation of (8.84) gives 
5 nn t = J 0*(t — nb)0(t — n'b)dt 

= ^- J dt j 0*(u))e-' lu,( -'- nb) du} J 4>{<J)& J(f - n ' v >dtJ 
= J dudu)'0*(u)0(J)e (un ~“' n ' )b 5(w-J) = J du\0(ui)\ 2 

00 p2ir(j+l')/b r2ir/b 00 

= V / du>\0(uj)\ 2 e i ^ n ~ n '' lb = / dw V \0(u + : 

j = -Jl*j/b Jo 

r2n/b 

= / drfMe-“ w . (8.103) 

Jo 

F(u>) is periodic with period £2q = 2n/b and can be represented as a Fourier sum 


\2 Auj(n—n')b 


- 2n j/b)\ 2 e~' u>A " h 


F{ui) = ^ Fn jn*n»ia* = ^ F„e 


i nbu 


(8.104) 


where the Fourier coefficients 

1 r® o u r2n/b 

F n = — / F(w)e~ l2 ™“ /r2 ° = — F(u)e~ mbul (8.105) 

FJq Jo 2n Jo 

are found from comparison with (8.103) 



(8.106) 
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Finally, evaluation of the Fourier sum (8.104) gives 

F(u) = T\0{uj + j^)\ 2 = (8.107) 

Mo 

j 

which is the equivalent of the orthonormality of @o„ in Fourier space. 

Equation 8.91 implies that @ m „can be represented as linear combination of the 
<P m+ 1 ,„. Starting from 

<p(0 = $o,o( 0 = ^h n 0 Un (t) = V2^h n <p(2t-nb) (8.108) 


scaling and translation gives 

n! 

Fourier transformation of (8.88) gives 


*m,n(u) = e- 2 '” ri "/ 12 "0 m ,o(tu) = e- 2 "- w /^0(tu/2 m ) 


V2™ 


^111+1,11 (tu) — 0 mn (uj/2) 

V2 


(8.109) 


( 8 . 110 ) 

( 8 . 111 ) 


with 

Q m = 2'" 2 y- = 2 m i2 0 
b 

and (8.108) becomes 


<p(uj) = ^ 

n 


h 

_ w — 2mriu)/Z2i 

V2 


<i>(uj/2 m ) = M 0 (u}/2)<l>(uj/2) 


( 8 . 112 ) 


(8.113) 


where 


M 0 (uj/2) = 


z 


,l n „-2nm(u//2)/a 0 

V2 


(8.114) 
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is periodic. Similarly, we find 


<P«o(w) =J^h„<P m+ lB (w) = X - 7 J= & - 2n ^ 2m+1 y^0(uj/2 m+1 ) 


V2'" +1 


Maiuj/2' n+x )<P(Lo/2 m+l ). 


sf2™ 


m+K 


(8.115) 


Equation 8.113 can be iterated to obtain 

<P(u) = M o (v/2)0{u>/2) = M o (o;/2)Mo(w/4)0(w/4) = 


= J]m 0 (u;/2'')<P(0) = Y[m 0 (u;/V)J — . 


7=1 


7=1 


(8.116) 


This equation shows that knowledge of M 0 is sufficient to determine the scaling 
function (see also p. 182). 

From the orthogonality condition (8.107) we obtain 

i- = ^|<j(cc + ;^o)| 2 = X M 0 (u/2 + j^pj 0^/2 + j^ 

= X |Mo(w/2 + jS2 0 )\ 2 \<P(u>/2 + j£2 o)| 2 
j 

+ 22 + ^./ + 2^ | ^ ^ w /2 + ^./ + 2^ | 

= |M o (o;/2)| 2 22|0( W /2 + y^o)| 2 


Mo(ut/2 + ^)| 2 X 


^((w + ^2o)/2 + 7 T2 q) |“ 


-L[ 

£2o 


— |M 0 (u;/2)| + 


Af 0 (w/2+^ | j 


(8.117) 


Example: Rectangular Scaling Function 

The simplest example of a scaling function is the rectangular function 


0(t) = 


75 fOT 

t - * 

' 2 

0 

else 


(8.118) 
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with the scaled and translated functions 


tf’o.i.O) = <P(t - nb) = 


yfb 


for 


f-{n+\)b 


<! 


0 else 


<P\ n {t) = \^20(2t — nb) — 


vm 


for 


? - (n + i) | 

0 else 


(8.119) 

( 8 . 120 ) 


< 2 y„(f) = V2™<P(2 m t - nb) 



for 



0 else 



( 8 . 121 ) 


Obviously, the & nw (t) for fixed m are orthonormal and can be represented as linear 
combination 


^m,n(0 — 


1 

7 ! 


< t > m+l,2n(t) + 


1 

7 ! 


^m+l,2n+l (0 * 


( 8 . 122 ) 


V m is the space of functions which are piecewise constant on intervals 1 1 — (n + 
\/2)b/2 m \ < b/2 m+1 . Figure8.16 shows the approximation of the parabola f(t) = 
t 2 by functions in Vo ... V 3 . 

The Fourier transform of the scaling function is 


<P(lu) = 


1 2 sin(f) 
~J2tt LO\fb 


e -iwfe/2 _ 



(8.123) 


Fig. 8.16 (Approximation 
by piecewise constant 
functions) The parabola 
f(t) = t 2 (dashed curve) is 
approximated by linear 
combination of orthonormal 
rectangular functions (8.121) 

fm(t) = Z„ Qmnit) 
f-oc Kn(0f(t)dt for 

in = 0 (black) m = 1 (red ) 
in = 2 (blue) in = 3 (green) 
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and from 


&(u>) 


1 2 [2 sin (f) cos (f)] 


yflTr 


Jy/b 




-iuiZ?/4 


(8.124) 


we find 

)-“' 4 


(8.125) 


5 . 6.2 Construction of an Orthonormal Wavelet Basis 

The approximation f (m+l> (t) contains more details than f (m] (t). We would like to 
extract these details by dividing the space 

V m+ i = V m + W m (8.126) 

into the sum of V m and an orthogonal complement W m _L V m . The approximation 
f 0n+l) (t) then can be divide into the approximation f (m> plus the projection onto 
W m , which provides the details. In the following we will construct an orthonormal 
basis of W m in terms of wavelet functions 'Pit) which have the properties 

(i) V e V m+1 (8.127) 

or 

<P = J^C„0 m+U n (8.128) 

n 

and 

(ii) 'P IV, (8.129) 


or 


= 0 'in 


which is equivalent to 

poo 

/ 'P*(u)<i> nt „(uj)duj — 0 in. 


(8.130) 


(8.131) 
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Fig. 8.17 (Haar wavelet) The rectangular scaling function (a) can be written as a linear combination 
of translated scaling functions at the next higher resolution (b). This is also the case for the wavelet 
function (c) which is orthogonal to the scaling function 


Example: Haar Wavelet 

With the rectangular scaling function 


<P(t) = 


1 if 0 < x < 1 
0 else 


the Haar wavelet [88] (Fig. 8. 17) 


(8.132) 


1 1 

<p(t) = —0i,o(0 - -=0i , O (0 

V2 V2 


(8.133) 


is a linear combination of the translated functions <P \and orthogonal to all 0o,«- 
The family of scaled and translated daughter wavelets 

1 1 

^m.a(f) — )=^m,/i(t) n-fl (f ) 

V2 V2 


obeys 


* m ,n e V m+l <P m . n _L V m . (8.134) 

Orthogonality Condition 

After Fourier transformation, (8.131) becomes 


0 = J 0 *(w)e-' !iw27r/s2 '"0 mO (w)rfw 
= X/ ^(w)e"“ 2l/C '4,o(^)^ 
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= X / " <P*(to + <P m0 (w + j£2 m )du> 


^—nvjj'l'K I Q tl 


Y', ip* (to + j ^2 m )0 m o(co + j£2 m )du> 


-niu2 »/fl- G(w ) du; = £2 m G(t n ). 


(8.135) 


This expression looks like the Fourier coefficient of an S2 m -periodic function with 
the Fourier sum (7.5 with to and t exchanged) 


A 27T 

G(uf) = ^ e‘'" w G(t ;! ) with t n = n—-. 




(8.136) 


But, since G(t) — 0, we obtain the orthogonality condition 

Y + 7«m)<Pmo(w + = 0 . (8.137) 

j 

Construction of the Wavelet 

Now, <Fand <P m o both are in V m+ \, therefore (8.113) 

&m0 = M mO (to/2 m+l )0(uj/2 m+l ) (8.138) 

<P = My(to/2 m+1 )0(to/2 m+1 ) (8.139) 

whei'e/V/„, o and My are S2q— periodic. 

Hence, from (8.137) 

0 = S M ^ ((w + jn m )/2 m+1 )M m0 ((u> + jQ m )/2 m+1 )\0((to + jS2 m )/2 m+l )\ 2 

j 

= Y K^/2 m+l + jn 0 /2)M m0 (to/2 m+l + jn o /2)\0(uj/2 m+l + jQ 0 /2)\ 2 

j 

= Y M*(cu/2 m+1 )M m0 (tu/2 m+1 2)|<f(a;/2 m+1 + G 0 j/2)\ 2 

j even 

+ YK (to/2 m+[ + M m 0 L/2 m+1 + \S(to/2 m+1 + S2 0 j/. 2)| 2 

j odd ' ' \ / 

= M^(L0/2 m+1 )M m0 (u>/2 m+l 2) Y \&(to/2 m+l +G 0 j/2)\ 2 

j even 

+ M* (at/2 m+1 + M m 0 L/2 m+1 + ^) X l^(W2 m+1 + ^ 0 7'/2)| 2 . 

' ' ' ' j odd 


(8.140) 
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From orthogonality of < P m +i. n 

T \<P(io/2 m+l + j£2o)\ 2 = (8.141) 

J 

we see that both sums have the same value 


T \0(io/2 m+1 + f2oj/2)\ 2 = V \<P(u;/2 m+1 + H2 0 )| 2 = i- (8.142) 

y even a: 

y |0(w/2 m+1 + a 0 j/2 )\ 2 = y |^(w/2 m+1 + ktt o + fl 0 /2 )| 2 = 

(8.143) 


and therefore 


M* (w/2 m+1 )M m0 (w/2 ,B+1 2) + M% (u>/2 m+l + ^ M„ )0 (w/2 m+1 + = 0 

(8.144) 


a icj27r/i2 wl+ i 


My ( ^/2 m+1 + ^ ) = M* i0 (uj/ 2 m+1 + ^o)e 


which can be satisfied by choosing [ 86 ] 

My (to/ 2 "’ +1 ) = M* 0 L/2 m+1 + e 1 

which implies 

? m+l . 

= —M* 0 (w/ 2 " ,+ 1 )e iw2,r/ ^'" +1 . 

Hence we obtain the solution 

^m(w) = e itW " +1 M* 0 L/2 m+1 + ^ <t>(u>/2 m+l ) 

h* 

— eL i(n'+\)u2'K/C2 m + x ^ ^,;/? w+ h 


(w/ 2 m+l + 5°) = 


i(u;+£2 m +i /2)27r/^2 m+ i 


(8.145) 


(8.146) 


(8.147) 


which becomes in the time domain 
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= T J= (-1)"'^L [ duj ut e i(n ' +l) “ 2 * / °" +1 <P(uj/2 m+1 ) 

“ J 

= ^(-l)"'/z*,V2 m + 1 0(2"’ +1 f + («' + \)b) 

n' 

= E(-i)"'^*- + w-i(o 

n' 

= Z(- 1 )“"“ 1 /l -n-l^+ I ."( f )- 

n 

From the orthogonality condition (8.107) we obtain 


(8.148) 

(8.149) 


y 1'7'otw + ;i2o)| 2 = ^ Moo f < 
j v 

= ^ M 0 0 


^0 

w/ 2 +a + D^ 


)| ®( w /2 + 7^) 


u>/2+(27 + 1)^ 


X M oo (w/2 + (2f + 2) 

(-/ 2 +f)fz 


f 2 o 


)IK 

)l>t 


-/2 + 2 2 f; 


w/2 + (2y + 1) 


12 o N 


= Moo 


|0( w /2 + 7 i 2 o )| 2 


+ |M 00 (oV2)| 2 X l*CC^ + ^o)/2 + 7^o)l 2 

= - | | Moo (u/2 + | + |Moo(w/2)|A . 


(8.150) 


But, since the scaling function obeys the orthonormality condition (8.107), 

~ 22 \ o)| 2 = 22 + j~2^ | ^ 2 + 7 —^ | 

= 22 l^oo(w/2 + j£2o)\ 2 \<P{ui/2 + j^)\- 

+ 22 ^°° + ^7 + | i ’ ^tu/2 + ^7 + -^ | 

= |M oo (tu/2)| 2 22|0(u;/2 + 7^o)| 2 


M»(w2+f) | Z 


|0((w + £ 2 q )/2 + 7^2o)|" 


[' 


= — | |Moo(w/2)| 2 + 


M 00 (u// 2 +^ | j 


(8.151) 


hence the wavelet also fulfills the orthonormality condition 
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T|^o(w + 7^o)| 2 = (8.152) 

Mq 
J 

Therefore the translated wavelet functions 

t - n2~ m b ) = ^(-l)-"'- 1 Ai„,_ 1 v/2^"0(2 m+1 (f 

n' 

= ^-^~ n '~ lh -n'-i et> ’n+l,2n+n'(t) 
n! 

are orthonormal 

j K n mnn'{t)dt = 5 n . n , (8.154) 

Wavelets for different resolution m are orthogonal since they are by construction 
in orthogonal spaces. The <£,„„(?) with m, n = —oo ... oo provide an orthonormal 
basis of 

OO 

L 2 (R)= U W„. (8.155) 


— n2 m b) — n'b) 

(8.153) 


Alternatively, (8.155) is replaced by 

oo 

L 2 (R) = y 0 + 1J w m (8.156) 

m =0 


which is more useful for practical applications with limited total observation time. 
According to (8.156), starting from a basic approximation in Vo, more and more 
details are added to obtain approximations with increasing accuracy. 

Example: Meyer Wavelet 

Meyer introduced the first non trivial wavelet (Fig. 8.18) which, in contrast to the Haar 
wavelet is differentiable [89, 90]. It was originally defined by its scaling function in 
Fourier space 11 (here, b = 1) 


<P(u>) = 


Tfe ifa; — T 


—i 

\/27T 


(f(¥-0) 

0 if I 


if — 

11 3 

4tt 
> 3 


< M 


4tt 
^ 3 


from which the mother wavelet can be derived 


(8.157) 


11 There are different variants of the Meyer wavelet in the literature. 
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3, 

c 

o 


Fig. 8.18 (Meyer wavelet in frequency space) Left scaling function Right magnitude of the wavelet 
function 


'P(uj) = 


i sin (;( : y - i )) c, ‘ n 1 

(f(f-i)) 

0 else. 


if 2 f < M < f 

-4= COS (? (- \ )) e 1 ^ 2 if % < ■' - g7r 


•n/27T 

1 

V27T 


M < f 


(8.158) 


Explicit expressions in the time domain (Fig. 8.19) were given in 2015 [91] 

4 


<p(t) = 


£ if? = ° 


sin f-t+U cos 


else 


(8.159) 




* 

C 

o 


Fig. 8.19 (Meyer wavelet in the time domain) Left scaling function Right wavelet function 
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# (t) = h ( r - 7) cos [f ( r - 5)] - l sin [f ( f - l)] 
, ^(r-l)cos[f(r-^)] + lsin[f(r-i)] 


(8.160) 


8.7 Discrete Data and Fast Wavelet Transform 

Mallet’s algorithm [87] starts with function values 

fn = .f(nAt s ) (8.161) 

sampled at multiples of 

At s = l/f, =b/2 m ~". (8.162) 

We do not really approximate the function but from the series of sample values we 
construct the linear combination 

(8-163) 

n 

which is an element of 

Mmax-l 

v >n mal = Vo + Q W m (8.164) 

m=0 


and can therefore be represented as a coarse approximation in Vo and a series of 
details with increasing resolution 

m max —l 

^ \ fn^m max ,n{ 0 ~ ^ ~f~ ^ ^ ' dmn ^ mn (/) ■ (8.165) 

n n m =0 « 


5.7./ Recursive Wavelet Transformation 

The approximation coefficients c„ and detail coefficients c/ m) , are determined recur¬ 
sively which avoids the calculation of scalar products. 

Starting with 

c m max 


,n = fn 


(8.166) 
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the details are extracted by expanding 

^ \ cm max ,n ( &m max ,n(t') = ^ ' Cnimax — 1 ,n& tn max — l,nit') "f" ^ ' dtn max —1 ,n ^m max — 1, n (0 • 
n n n 

(8.167) 

Due to orthogonality, the coefficients at the next lower resolution can be determined 
from 

Ctn max — l,n' — ^ v ^>n max ,n ^tn max — fw' I ^m max ,n - > 
n 

~~ ^ ^ Cm max ,n+2n'h n 
n 

^m m ax — 1 jW 7 = ^ , Cfflmax,n ^ ^Wffiax — 1»« / I ^W majc ,« " > 
n 


— y~! c m maXl nh n 


-In' 


(8.168) 




n —1 


(8.169) 


which can be written as 

= Y. C m ^g*n- 2»' = Z C nimax ,n+2n' 9 n with g* = (-1)" 

n n 

(8.170) 

Iterating this recursion allows the calculation of the wavelet coefficients even with¬ 
out explicit knowledge of the scaling and wavelet functions. Equations (8. 168) and 

(8.170) have the form of discrete digital filter functions with subsequent downsam¬ 
pling by a factor of two (dropping samples with odd n')} 2 This can be seen by 
defining the down sampled coefficients 

4/2 = Z^«-«' ( 8 - 171 ) 

n 

4,2 = (8-172) 

and applying the z-transform to (8.168) and (8.170). For the approximation filter we 
obtain 


12 For the more general class of bi-orthogonal wavelets, a different filter pair is used for reconstruc¬ 
tion. 
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oo oo 


fciz) = X Z KCn+n'Z-"' 


n'——oo n=—o o 


= Z h >" Z = (z h ^~ n ) Z c *' z ~ n ' = 

n n' \ n / n' 


= h\z)c(z) (8.173) 

hence in frequency space the signal is multiplied with the filter function 

h{e lu>At ) = 'Y_ l h n Q- niuA ' = V2M 0 (u>). (8.174) 

n 

Similar we obtain for the detail filter 

fd(z) = Z^C-l^'An'-n-lZ - "' = Z c n z- n (-l) n - l h n ’-n-izr- n '. (8.175) 

nn' 

Since only even values of n’ are relevant, we may change the sign by (—1)" to obtain 

Z c »z“"(-l)"'“" _I/j »'-"-i^“"' = z*H-z) c{z) = g*(z) c(z) (8.176) 

where 

9(z) = z ^r~ x h*_ n _ x z- n = Zt- 1 )— 2 ^ 1 = z z *:(-*)- 


= z(Z / '»(- J )'") = zh*(—z). 


(8.177) 


7.2 Example: Haar Wavelet 


For the Haar wavelet with 


13 


/ 7 0 = h\= —= h„ — 0 else 

V2 


(8.178) 

(8.179) 


13 The standard form of the Haar wavelet with go = l/\/2, g\ = —l/y/2 differs from (8.179) by a 
shift and time reversal. The resulting wavelet basis, however, is the same. 
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Fig. 8.20 Haar filter pair 



we obtain the filter functions 

h{z) = Ti (‘ + 0 = 


(8.180) 


On the unit circle, 


|/t(e lt ' Mf )| 2 = 1 + cos co At 

(8.181) 

\g(e lulA, )\ 2 = 1 — cost oAt 

(8.182) 


which describes a low and a high pass forming a so called quadrature mirror filter 
pair (Fig. 8.20) [92], 


8.7.3 Signal Reconstruction 

The wavelet transformation can be inverted using the expansion 

C m,n C t > m,n(t) = z Cm— l,n^m— l,n(0 “1“ ^ ^ d m — \,n l,n (0 (8.183) 

n n n 

where the coefficients at the higher level of approximation are obtained from 

Cm,n' — ^ ' Cm—l,n < ' l,n ^ ^ —1,« < &m,n' I 1,« - > 

n n 

— ^ ' C m —i n h n '—2n ^ ^ d m —\ n ( 1) ^2n— n '—1 

= ^ ^ C m —\ n h n '—2n “1" ^ 7 d m —\ n ( 1) ^2n—1’ 

n n 


(8.184) 
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This can be formulated as upsampling and subsequent filtering. Formally, we insert 
zeros and define the up sampled coefficients 


c 2n — c m-l,n c 2n+l ~ ® 

(8.185) 

d 

II 

+ 

dr* 

^3 

K 

7 

II 

^3 

(8.186) 

Then, 


^ \ ^m—\,nhn'—2n = ^ ' ^2n^ n '~^ n = ^ 1 ^ n ^n’—n 

(8.187) 

n n 


= (-1 )"'-* Y, d lK_ n ,_ 1 = X(- VTdngrV-n 

n 

(8.188) 

where due to (8. 186) the alternating sign can be omitted. Z-transformation then gives 

'Y^CnK'-nZ = ^C„Z ”h n ,_ n Z” = h(z) c(z) 

(8.189) 


n,n' n,n' 


Y.dndn'-n = 9(z)d(z ) = zh*(- Z )d( Z ). (8.190) 

nn' 


8.7.4 Example: Analysis with Compactly Supported Wavelets 


Wavelet analysis has become quite popular for processing of audio and image data. 
In Problem 8.3 we use Daubechies wavelets [93] to analyze a complex audio signal 
consisting of a mixture of short tones, sweeps and noise (Figs. 8.23,8.24). Daubechies 
satisfies (8.117) by taking 


M 0 (cu/2) = 


1 

2 


N 


(1 + e -i “/ 2 ) 


0(e- iw/2 ) 


(8.191) 


with a trigonometric polynomial Q. This leads to a class of compactly supported 
orthonormal wavelet bases, which for N = 1 include the Haar wavelet as the simplest 
member. For N = 2, 


Mq{lo/2) = 


5 < 1+ ' 


■ /2 >T|[('+^)+(i-^) 


-\uj/2 


(8.192) 
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= ^ [(i + V3) + ( 3 + V3) e“ iw/2 + ^3 - 73) e" 2iw/2 + (l-V3j e“ 3iw/2 ] 

(8.193) 


with the four nonzero scaling parameters 


V2 r~ 

h 0 = ^-(1 + V3)« 

0.48296 

(8.194) 

V2 r 

h\ = — (3 + V3)^ 

O 

0.83652 

(8.195) 

V2 n 

h 2 = ^—(3 - V3) « 

0.22414 

(8.196) 

V2 r~ 

h 3 = — (1 - V3) « 

-0.12941. 

(8.197) 

This defines the wavelet basis which is known as Daubechies 2. There are no analytic 
expressions for the scaling and wavelet functions available. They can be calculated 
numerically from the infinite product (8.116) or a corresponding (infinitely) nested 
convolution in real space. Figures 8.21 and 8.22 show the fast convergence. 



Fig. 8.21 (Daubechies 2 scaling function) The scaling function is calculated numerically in the 
time domain from the Fourier transform of (8.116) with a finite number of factors. The blue curve 
shows the result for j max = 7, red dots show results for j max = 5, black dots for j max = 3- Delta 
functions are replaced by rectangular functions of equal area 
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2 



-2 


-1 


0 


2 


time t 


Fig. 8.22 (Daubechies 2 wavelet function) The wavelet function is calculated numerically in the 
time domain from the Fourier transform of (8.116) and (8.147) with a finite number of factors. 
The blue cur\<e shows the result for j max = 7, red dots show results for j max = 5, black dots for 
jmax = 3. Delta functions are replaced by rectangular functions of equal area 


Problems 

Problem 8.1 Short Time Fourier Transformation 

In this computer experiment STFT analysis of a frequency modulated signal 


f(t) — sin 0(f) = sin + 
with a momentaneous frequency of 



(8.198) 


80 

ui(t) = —- = cjq(1 + a sinuqf) 


(8.199) 


is performed and shown as a spectrogram (Figs. 8.10, 8.11). Sampling frequency is 
44100Hz, number of samples 512. 

You can vary the carrier frequency uiq, modulation frequency u>i and depth a as 
well as the distance between the windows. Study time and frequency resolution 

Problem 8.2 Wavelet Analysis of a Nonstationary Signal 

In this computer experiment, a complex signal is analyzed with Morlet wavelets over 
6 octaves (Fig. 8.14). The signal is sampled with a rate of 44 kHz. The parameter d of 
the mother wavelet (8.61) determines frequency and time resolution. The frequency 
coo of the mother wavelet is taken as the Nyquist frequency which is half the sampling 
rate. The convolution with the daughter wavelets (8.76) is calculated at 400 times 
with a step size of 0.726 ms (corresponding to 32 samples) 
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t n — to + nAt 


( 8 . 200 ) 


and for 300 different values of the scaling parameter 
s m = 1.015'". 


( 8 . 201 ) 


The signal consists of two sweeps with linearly increasing frequency of the form 

/ 1 . 2 (f) = sin t 2 ] (8.202) 

and another component which switches between a 5 kHz oscillation and the sum of 
a 300 Hz and a 20 kHz oscillation at a rate of 20 Hz 


hit) = 


sin(w 2 o kHzt) + sin(w 3 oo h-J) if sin(cj 2 or/H) < 0 
sin(tU 5 A:/f,t) else. 


(8.203) 


Study time and frequency resolution as a function of d 

Problem 8.3 Discrete Wavelet Transformation 

In this computer experiment the discrete wavelet transformation is applied to a com¬ 
plex audio signal. You can switch on and off different components like sweeps, dial 
tones and noise. The wavelet coefficients and the reconstructed signals are shown, 
(see Figs. 8.23, 8.24). 


c 

’o 

<+-t 

<D 

o 

o 






time (sec) 


Fig. 8.23 (Wavelet coefficients of a complex audio signal) From Top to Bottom The black curve 
shows the input signal. The finest details in light green , red and blue correspond to a high fre¬ 
quency sweep from 5000-15000 Hz starting at 0.7 s plus some time dependent noise. Cyan, orange 
and maroon represent a sequence of dial tones around 1000 Hz, dark green and magenta show 
the signature of several rectangular 100 Hz bursts with many harmonics. The black curi’e at the 
Bottom shows the coefficients of the coarse approximation, which essentially describes random 
low frequency fluctuations. The curves are vertically shifted relative to each other 
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. 



■ < ■ X ■- 



0 0.5 1 1.5 

time (sec) 


Fig. 8.24 (Wavelet reconstruction) The different contributions to the signal are reconstructed from 
the wavelet coefficients. Color code as in Fig. 8.23. The original signal ( Top black cun>e ) is exactly 
the sum of the coarse approximation (Bottom black curve) and all details (colored curves). The 
curves are vertically shifted relative to each other 














Chapter 9 

Random Numbers and Monte-Carlo 
Methods 


Many-body problems often involve the calculation of integrals of very high dimension 
which can not be treated by standard methods. For the calculation of thermodynamic 
averages Monte Carlo methods [94-97] are very useful which sample the integration 
volume at randomly chosen points. In this chapter we discuss algorithms for the 
generation of pseudo-random numbers with given probability distribution which are 
essential for all Monte Carlo methods. We show how the efficiency of Monte Carlo 
integration can be improved by sampling preferentially the important configurations. 
Finally the famous Metropolis algorithm is applied to classical many-particle systems 
and nonlinear optimization problems. 


9.1 Some Basic Statistics 

In the following we discuss some important concepts which are used to analyze 
experimental data sets [98]. Repeated measurements of some observable usually 
give slightly different results due to fluctuations of the observable in time and errors 
of the measurement process. The distribution of the measured data is described by a 
probability distribution, which in many cases approximates a simple mathematical 
form like the Gaussian normal distribution. The moments of the probability density 
give important information about the statistical properties, especially the mean and 
the standard deviation of the distribution. If the errors of different measurements 
are uncorrelated, the average value of a larger number of measurements is a good 
approximation to the “exact” value. 


9.1.1 Probability Density and Cumulative Probability 
Distribution 

Consider an observable £, which is measured in a real or a computer experiment. 
Repeated measurements give a statistical distribution of values. 
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Fig. 9.1 (Cumulative 
probability distribution of 
transition energies) The 
figure shows schematically 
the distribution of transition 
energies for an atom which 
has a discrete and a 
continuous part 



The cumulative probability distribution (Fig. 9.1) is given by the function 
Fix) = P{£ < x) (9.1) 

and has the following properties: 

• F(x) is monotonously increasing 

• F(—o o) = 0, F(o o) = 1 

• F(x) can be discontinuous (if there are discrete values of £) 

The probability to measure a value in the interval x\ < £ < x 2 is 

P(x i <£,< x 2 ) = F(x 2 ) - Fix i). (9.2) 

The height of a jump gives the probability of a discrete value 

P(£ = xo) = F(xo + 0) - F{x o - 0). (9.3) 

In regions where F(x) is continuous, the probability density can be defined as 

/(x 0 ) = F\x o) = lim -)-P(xq < ^ < x 0 + Ax). (9.4) 

Av—>0 Ax 


9.1.2 Histogram 

From an experiment F(x) cannot be determined directly. Instead a finite number N 
of values x, are measured. By 


Z N {x) 
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Fig. 9.2 (Histogram) The 
cumulative distribution of 
100 Gaussian random 
numbers is shown together 
with a histogram with bin 
width Ax = 0.6 



x 


we denote the number of measurements with Xi < x. The cumulative probability 
distribution is the limit 

F(pc) = lim ]-Z N (x). (9.5) 

N^oo N 

A histogram (Fig. 9.2) counts the number of measured values which are in the interval 

Xi < x < x i+i : 


^(Z N {x i+ 1 ) - Z N (x,)) « F(x i+ 1 ) - F{Xi) = P{Xi < £ < x i+ i). (9.6) 

Contrary to Z N (x) itself, the histogram depends on the choice of the intervals. 


9.1.3 Expectation Values and Moments 


The expectation value of the random variable £ is defined by 

/ oo pb 

xdF{x) = lim / xdF(x) 

-oo a ^ oo,b ^oo J a 

with the Riemann-Stieltjes-Integral [99] 


xdF(x)= lim ^ Xj{F{Xj) - F{Xj- i))| J . =a+ g = a,-. 

N^oeX-~i i v 

i= 1 


(9.7) 


(9.8) 


Higher moments are defined as 
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£[£*] = / x k dF(x) 


(9.9) 


if these integrals exist. Most important are the expectation value 
* = £[£] 

and the variance, which results from the first two moments 

a 2 — J (x — x) 2 dF = J x 2 dF + J x 2 dF — 2 x J xdF 
= E[f] - (£[£]) 2 - 


(9.10) 


(9.11) 


The standard deviation a is a measure of the width of the distribution. The expectation 
value of a function < p(x) is defined by 

/ OO 

ip(x)dF(x). (9.12) 

-OO 

For continuous F(x ) we have with dF(x) = f(x)dx the ordinary integral 


£[f] = / x k f(x)dx 


E[ip(x)] - / ip(x)f(x)dx 


(9.13) 

(9.14) 


whereas for a pure step function F(x) (only discrete values x; are observed with 
probabilities p(x,) = F(x/ + 0) — F(x t — 0)) 


E[H k ] = 

E[ip{x)] = y, ip(Xj)p(Xj). 


(9.15) 

(9.16) 


9.1.4 Example: Fair Die 

When a six-sided fair die is rolled, each of its sides shows up with the same probability 
of */6. The cumulative probability distribution F(x) is a pure step function (Fig. 9.3) 
and 


' 6 1 J 1 21 

XdF = y Xj{F(Xj + 0) - Fix, - 0)) = - y x, = — = 3.5 


(9.17) 


1=1 


1=1 
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Fig. 9.3 Cumulative 
probability distribution of a 
fair die 


0 1 2 3 4 5 6 X 


_ ® 1 6 91 

* 2 = 2>i 2(F(x '+°) - F &i - °» = g = -g- = 15 - 1666 • • • 

i= 1 i= 1 

(7 = yjx 2 —X 2 — 2.9. 


(9.18) 

(9.19) 


9./.5 Normal Distribution 

The Gaussian normal distribution is defined by the cumulative probability distribution 


i r x 

<P{x) = —= / e~‘ 2/2 dt (9.20) 

V 2,77 J —oo 

and the probability density 

<p(x) = -2 =q~ x2/1 (9.21) 

V27T 

with the properties 

/ oo 

(p(x)dx = 0(oo) = 1 (9.22) 

-OO 

/ OO 

x(f(x)dx = 0 (9.23) 

-oo 

/ oo 

x 2 tp(x)dx — 1. (9.24) 

-OO 
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Since 0 (0) = \ and with the definition 

<P 0 (x) = — L= [ e~‘ 1/2 dt 
V27T Jo 

we have 

1 

®(x) = -+ 0 o (x) 

which can be expressed in terms of the error function 1 

2 r x 

erf(.r) = —— / e~' 2d, dt = 2<Pq(V2x) 

Jo 

as 

0O(X) = 


(9.25) 


(9.26) 


(9.27) 


(9.28) 


A general Gaussian distribution with mean value x and standard deviation a has the 
probability distribution 



and the cumulative distribution 



(9.29) 


(9.30) 

(9.31) 


9.1.6 Multivariate Distributions 

Consider now two quantities which are measured simultaneously. £ and 77 are the 
corresponding random variables. The cumulative distribution function is 

F(x,y) = P(£ < x and rj < y). (9.32) 


1 erf Or) is an intrinsic function in FORTRAN or C. 
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Expectation values are defined as 


E[<p(x, >’)] = 


(p(x, y)d 2 F(x, y). 


' —oo J — oo 


For continuous F(x,y) the probability density is 

d 2 F 

fix, y) = 


dxdy 

and the expectation value is simply 

/ OO p OO 

dx / dy<p(x,y)f(x,y). 

-oo J — OO 

The moments of the distribution are the expectation values 
M kJ = Etfrf]. 

Most important are the averages 
* = £[£] y = E[rj] 


and the covariance matrix 

/ E[(f - x) 2 ] £[(£ - x)fq - y)] x 2 -x 2 xy- xy 

\E[^-x)(V-y)] E[( 1 7 -y) 2 ] J \xy-xyy 2 -f 

The correlation coefficient is defined as 

xy — xy 

If there is no correlation then p — 0 and F(x, y) = F\ (x ) F 2 (y ). 


(9.33) 


(9.34) 


(9.35) 


(9.36) 


(9.37) 


(9.38) 


(9.39) 


9.1.7 Central Limit Theorem 

Consider N independent random variables £,■ with the same cumulative distribution 
function F(x), for which £[£] — 0 and F[ k 2 ] = 1. Define a new random variable 

6 + £2 + • • • Civ 

Vn 


m = 


(9.40) 
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Fig. 9.4 (Central limit 
theorem) The cumulative 
distribution function of r) 
(9.42) is shown for N = 4 
and compared to the normal 
distribution (9.20) 



11 


with the cumulative distribution function F N (x). In the limit N -» oo this distribution 
approaches (Fig. 9.4) a cumulative normal distribution [100] 

1 f' x 

lim F n (x) = <P(x) = _ / eT'! 2 dt . (9.41) 

N ^°° V27T J -oo 


9.1.8 Example: Binomial Distribution 


Toss a coin N times giving £,■ = 1 (heads) or & — — 1 (tails) with equal probability 
P = Then £[£,] = 0 and F[E^] = 1. The distribution of 



Z6 


(9.42) 


can be derived from the binomial distribution 


1 = 


1 

2 



2- X (-D"- 

p =o 



(9.43) 


where p counts the number of tosses with £ = +1. Since 


n = p ■ 1 + (N — p) • (— 1) = 2p — N e [-N, N] 


(9.44) 
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the probability of finding ij = is given by the binomial coefficient 


P(r\ = 2P r J f ) = 2 ~ n 


Vn 


( n ~p ) 


or 


n m 

P(v = -=) = 2~ N 

Vn 


(A). 


(9.45) 


(9.46) 


9.1.9 Average of Repeated Measurements 

A quantity X is measured N times. The results X\ ■ ■ ■ X\ are independent ran¬ 
dom numbers with the same distribution function /(X,). Their expectation value 
is the exact value £[X,] = f dXjX/f(X,) = X and the standard deviation due to 

measurement uncertainties is ax = E[X ( 2 ] — X 2 . The new random variables 
Xi-X 

6 = -- (9-47) 

<Jx 

have zero mean 


E[X t ] - X 

e [ ei = -= 0 


and unit standard deviation 


(9.48) 


<*1 = £[£, 2 ] - Etff = E 


X f+X 2 - 2XX -, 


Hence the quantity 

Z U< Z Ui-N x VN- 

\f~N V/V ax 
obeys a normal distribution 


E[Xf] - X- 


= 1. 


= — (X-X) 
<Jx 


f(v) = 


1 


*~rf/ 2 


\Z2ty 


(9.49) 


(9.50) 


(9.51) 
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From 


f (X)dX = f(rj)dr] =f(j](X ))- dX 


(9.52) 


we obtain 



(9.53) 


The average of N measurements obeys a Gaussian distribution around the exact 
value X with a reduced standard deviation of 



(9.54) 


9.2 Random Numbers 

True random numbers of high quality can be generated using physical effects like 
thermal noise in a diode or atmospheric noise [101]. Computers very often make 
use of pseudo random numbers which have comparable statistical properties but 
are not totally unpredictable. For cryptographic purposes sophisticated algorithms 
are available which are slow but cryptographically secure, e.g. the Yarrow [102] 
and Fortuna [103] algorithms. In computational physics, usually simpler methods 
are sufficient which are not cryptographically secure, but pass important statistical 
tests like Marsaglia’s DIEHARD collection [104, 105] and TestUOl [106, 107], 
Most methods use an iterated function (Sect. 22.1). A set of numbers Z (e.g. 32-bit 
integers) is mapped onto itself by an invertible function/(r) and, starting from a 
random seed number ro e Z, the sequence 


n+i =f(n) 


(9.55) 


is calculated to provide a series of pseudo random numbers [105]. Using 32-bit 
integers there are 2 32 different numbers, hence the period cannot exceed 2 32 . The 
method can be improved by taking Z to be the set of m-tuples of 32-bit integers 
r = {z\, Z2 ■ ■ ■ Z m } and f(r) a function that converts one m-tuple into another. An 
m-tuple of successive function values defines the iteration 

n = {zi, Zi -i - Zi- m + i } (9.56) 


A I I — [m'+Ij Zi, ■ • • Zi—m+ 2 } — {f (Zi , ■ • • Zi—m+ l)j Zi, ... A—m+21 ■ 


(9.57) 
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Table 9.1 Addition modulo 2 
0 + 0 = 0 
1 + 0=1 
0+1 = 1 
1 + 1=0 


Using 32-bit integers, (9.57) has a maximum period of 2 32 '” (Example: for m = 2 
and generating 10 6 numbers per second the period is 584942 years). For the initial 
seed, here in independent random numbers have to be provided. 

The special case of a lagged RNG simply uses 

+ ■ I = {z*+l , Zi, • ■ • Zi— m+ 2 } = {f iZi— »;+l), Zi , . . . Zi— m+2 } - (9.58) 

Popular kinds of functions fir) include linear congruent mappings, xorshift, 
lagged Fibonacci, multiply with carry (MWC), complimentary multiply with carry 
(CMWC) methods and combinations of these like the famous Mersenne Twister 
[108] and KISS [105] algorithms. We discuss briefly some important principles. 


9.2.1 Linear Congruent Mapping (LC) 

A simple algorithm, mainly of historical importance due to some well known 
problems [109], is the linear congruent mapping 


r i+ i = (art + c) mod b (9.59) 

with multiplier a and base b which is usually taken to be b = 2 32 for 32-bit integers 
since this can be implemented most easily. The maximum period is given by b. 


9.2.2 Xorshift 

A 32-Bit integer 2 can be viewed as a vector r = (bo, b\ ... /+ ]) of elements b, in the 
field T-i = [0, 1}. Addition of two such vectors (modulo 2) can be implemented with 
the exclusive-or operation as can be seen from comparison with the table (Table 9.1). 

An invertible linear transformation of the vector r can be described by multipli¬ 
cation with a nonsingular 32 x 32 matrix T 


/(r) = r T. 


(9.60) 


2 This method can be easily extended to 64-Bit integers. 
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Fig. 9.5 Multiply with carry 
method using 64-bit integer 
arithmetic 



To simplify the numerical calculation, Marsaglia [105] considers matrices of the 
special form 3 

T = (1 +L a )(l +R b )(l +L C ) (9.61) 

where L ( R ) is a matrix that produces a left (right) shift by one. For properly chosen 
numbers a, b, c the matrix T is of order 2 32 — 1 and the random numbers have the 
maximum possible period. There are many possible choices, one of them leads to 
the sequence 

y = y xor (y «; 13) 

y = y xor(y » 17) 

y = yxor(y <£ 5). (9.62) 


9.2.3 Multiply with Carry (MWC) 

This method is quite similar to the linear congruent mapping. However, instead of 
the constant c in (9.59) a varying carry is used. 

For base b = 2 32 and multiplier a — 698769069 consider pairs of integers r — 
[x, c] with 0 < c < «, 0 < x < b excluding [0, 0]and [a — 1, b — 1] and the 

iteration function 4 

/([x, c]) = [ax + c mod b, (ax + c)/b]. (9.63) 

Starting with a random seed [xo, co] the sequence [x^, c*] =/([x*_i, c*_i]) has 
a period of about 2 60 [105], If one calculates t — ax ,t + q in 64 bits, then for 
b — 2 32 ,q + i is given by the top 32 bits and x^+iby the bottom 32 bits (Fig. 9.5). 


3 At least three factors are necessary for 32 and 64-Bit integers. 

4 Using integer arithmetics. 
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9.2.4 Complementary Multiply with Carry (CMWC) 

The simple MWC method has some inherent problems which can be overcome by a 
slight modification. First, the base is taken to be b = 2 32 — 1 and second the iteration 
is changed to use the (b — l)-complement 


x k — (b — 1) — (axk -i + c k — i) mod b. (9.64) 

This method can provide random numbers which pass many tests and have very large 
periods. 


9.2.5 Random Numbers with Given Distribution 

Assume we have a program that generates random numbers in the interval [0,1] like 
in C: 


rand () / (double)RAND_MAX. 


The corresponding cumulative distribution function is 


FoU) 


H for x < 0 
x for 0 < x < 1 . 
1 for x > 1 


(9.65) 


Random numbers with cumulative distribution F(x) can be obtained as follows: 

choose a RN r e [0, 1] with P(r < x) — Fq(x) 
let f — F~ 1 (r) 

F(x) increases monotonously and therefore 


P(f < x) = P(F(0 < F{x)) = P(r < F(x)) = F 0 (F(x)) (9.66) 


but since 0 < Fix) < 1 we have 


P(f < x) = Fix). 


(9.67) 


This method of course is applicable only if F 1 can be expressed analytically. 
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9.2.6 Examples 

9.2.6.1 Fair Die 


A six-sided fair die can be simulated as follows: 


choose a random number r e [0, 1] 

1 for 0 < r < i 


Let £ = F~ l (r) = 


for 

for 

for 

for 

for 


l*r<l 

Ur<\ 

l<r<t 

i<r<l 

I < r < 1 


9.2.6.2 Exponential Distribution 

The cumulative distribution function 
F(x) = 1 - e~ x/x 

which corresponds to the exponential probability density 

fix) = je~ x/X 

can be inverted by solving 
r = 1 - e~ x/x 


for x: 

choose a random number r e [0, 1] 

Let .r = F~ l (r) = — A In(1 — r). 

9.2.6.3 Random Points on the Unit Sphere 

We consider the surface element 

1 , 

— R~d(p sin 9d0. 

An 


(9.68) 


(9.69) 


(9.70) 


(9.71) 


Our aim is to generate points on the unit sphere (9, ip) with the probability density 
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f(0, p)dpd9 = — dip sin 9d6 — - dtpd cos 0. (9.72) 

47r 47T 

The corresponding cumulative distribution is 

i r ost> r cp i — cos o 

F(6,ip) = -— d cos 6 I dip=^~ --- =F lp F e . (9.73) 

4tt J i Jo 2tt 2 

Since this factorizes, the two angles can be determined independently: 

choose a first random number r\ e [0, 1] 

Let ip = F-Vi) = 27rn 

choose a second random number n e [0, 1] 

Let 9 — Fg l (r 2 ) = arccos(l — 2rF) 


9.2.6.4 Gaussian Distribution (Box Muller) 


For a Gaussian distribution the inverse F ~ 1 has no simple analytical form. The 
famous Box Muller method [110] is based on a 2-dimensional normal distribution 
with probability density 


fix, y) = — exp 


x 2 +y 2 
2 


(9.74) 


which reads in polar coordinates 


1 2/9 

fix, y)dxdy = f p (p, p)dpdip — e~ p ' pdpdip. 

2tt 


(9.75) 


Hence 


f p (p, ip) = ^-pe p 2/2 (9.76) 

Z7T 

and the cumulative distribution factorizes: 

F„(p, <p) = ^cp- [ P p'e-^ l2 dp' = ^(1 - e _p2 ) = F p (p)F p {p). (9.77) 
The inverse of F p is 
P = J- ln(l - r) 


(9.78) 
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and the following algorithm generates Gaussian random numbers: 

rl=RN e [0, 1] 
r2 = RN e [0, 1] 
p = V- ln(l - n) 
p = 2irr2 
x = p COS ip. 


9.3 Monte-Carlo Integration 

Physical problems often involve high dimensional integrals (for instance path inte¬ 
grals, thermodynamic averages) which cannot be evaluated by standard methods. 
Here Monte Carlo methods can be very useful. Let us start with a very basic 
example. 


9.3.1 Numerical Calculation of it 

The area of a unit circle (r = 1) is given by r 2 n = it. Hence 7r can be calculated by 
numerical integration. We use the following algorithm: 

choose N points randomly in the first quadrant, for instance N independent 
pairs x, y e [0, 1] 

Calculate r 2 = x 2 + y 2 

Count the number of points within the circle, i.e. the number of points 
Z(r 2 < 1). 

| is approximately given by Z( ' A ,~ 1) 

The result converges rather slowly (Figs. 9.6, 9.7). 


9.3.2 Calculation of an Integral 

Let £ be a random variable in the interval [a, b] with the distribution 

]f- a for xe [ a , b] 


P(x < £ < x + dx) = f(x)dx = 


0 else 


(9.79) 


The expectation value of a function g(x) is 


E[g(x)] = / g(x)f{x)dx = / g(x)dx 


(9.80) 
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Fig. 9.6 Convergence of the 
numerical integration 



400 600 

N 


1000 


Fig. 9.7 Error of the 
numerical integration 



hence the average of N randomly taken function values approximates the integral 


l N _ 

g(x)dx <*> - ^ 5 ( 6 ) = g(0- 

V i=l 


(9.81) 


To estimate the error we consider the new random variable 


1 N 


N . , 
1= 1 

Its average is 


(9.82) 


7 = £[7] 


1 J , 

— ^E[g(x)] = E[g(x )] 



g(x)dx 


(9.83) 
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and the variance follows from 

= E Vil - 7) 2 ] = E l^H 5(6) - 7) 2 ] = E[{^ X ( 5(6) - 7)) 2 ] (9-84) 

- ^2 £ [Z(5(6) - 7) 2 ] = ±W&~W) 2 ) = ^ 2 (0- (9-85) 

The width of the distribution and hence the uncertainty falls off as 1 / */N. 


9.3.3 More General Random Numbers 


Consider now random numbers £ e [a, b] with arbitrary (but within [a, b] not 
vanishing) probability density/(.r). The integral is approximated by 


ly ^ = E 


g(x) 

.fix) 


jp-f(x)dx= I g(x)dx. 
3 fix) J a 


The new random variable 


T = 


J_ y ff(6) 


(9.86) 


(9.87) 


according to (9.85) has a standard deviation given by 
1 ,9iO, 

0Y = —^cr(——) 

Vn fiO 


(9.88) 


which can be reduced by choosing/ similar to g. Then preferentially £ are generated 
in regions where the integrand is large (importance sampling). 


9.3.4 Configuration Integrals 

Consider a system which is described by a ndim dimensional configuration space 
q i ... q m nm where a certain configuration has the normalized probability density 

Qiqu ■ ■ ■ Qndim) (9.89) 


Qiqi - qnd,m)dq'“ l,m = 1. 


(9.90) 
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The average of an observable A{q\ ... q„di m ) has the form 

< A > = I... I A(qi... q ndi m)Q{qi , ■ ■ ■ q ndm )dq ndm (9.91) 

which will be calculated by MC integration. 

Classical Thermodynamic Averages 

Consider a classical N particle system with potential energy 


V(q i ■ • - qm)- 


(9.92) 


The probability of a certain configuration is given by its normalized Boltzmann-factor 


Q-PV(q\...q iN ) 

e(qi 


(9.93) 


and the thermal average of some observable quantity A{q\ ... q^) is given by the 
configuration integral 


< A > = / A(q x ... q„d,m)Q(q\ ■ ■ ■ q3N)dq 


3 N 


f dq 3N A( qi ... q ndim )e-^ -^ 
Jdq 3N e-P v te i-?w) 


(9.94) 


Variational Quantum Monte Carlo method 

Consider a quantum mechanical N particle system with Hamiltonian 


H=T+V(qi. ..qw)- 


(9.95) 


According to Ritz’s variational principle, the ground state energy is a lower bound 
to the energy expectation value of any trial wavefunction 


^trial | ^trial ^ 


(9.96) 


Energy and wavefunction of the ground state can be approximated by minimizing 
the energy of the trial wavefunction, which is rewritten in the form 
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E __ f %*ial(q 1 ■ ■ ■ ■ ■ ■ q3N)dq M 

V J Wtrialiqi ■ ■ ■ q3N)\ 2 dq 3N 

_ f g(qi ■ ■ ■ q3 N )E L (qi ■ ■ • qw)dq m 
f B(qi ■ ..qi N )dq iN 

with the probability density 

g(qi ... t/ 3 /v) = \&tnai(q\ ■ ■ ■ <73iv)|- 

and the so called local energy 

„ W/naliq] ■ ■ ■ 43/v) 

E L = -• 

'Etrial (qi ■ ■ ■ <?3iv) 


(9.97) 


(9.98) 


(9.99) 


9.3.5 Simple Sampling 


Let £ be a random variable which is equally distributed over the range r/ nlln • • • c/ max , 
i.e. a probability distribution 


P(£, e[q.q + dq ]) =f(q)dq 


f(q) = 


f (q)dq = 1 . 


q e[q mini r/ m ax] 

0 else 


(9.100) 

(9.101) 

(9.102) 


Repeatedly choose ndim random numbers <^{ m) , ... £,^ im and calculate the expectation 
value 

1 M 

E (A(^l • • • indim)Q (£,\- • • • £ndim )) = ^ A(t;[ . . . ) • • • 

m= 1 


— J A(C[\ . . . Qndim) • • • Qndim)f (*Zl) * * ' f i^Qndim)^Q\ * * ' ^Qndim 

J r *7 max f ^max 


(n —n \ndim 
V^max *2 min/ 


MVl • • • Qndim)Q(S{\ • • • Qndim)dci 


ndim 
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Hence 

^ (A(£i * * * £,ndim)i • • • £ ndim )) 

E (£?(£l 5 ■ • • £,ndim )) 

/?r • • • ZT A ^' ■ ■ ■ <lndim)Q{qi ■ ■ ■ qndim)dq ndim 


(9.103) 


Each set of random numbers defines one sample configuration. The 

average over a large number M of samples gives an approximation to the average 
< A >, if the range of the q, is sufficiently large. However, many of the samples will 
have small weight and contribute only little. 


9.3.6 Importance Sampling 

Let us try to sample preferentially the most important configurations. Choose the 
distribution function as 

f (q I ‘ ‘ ■ qndim ) — Q6l\ • • ■ qndirn) • (9.104) 

The expectation value of A now directly approximates the configurational average 

1 M 

E (A(6 • • • U,)) = Jim - X -MtF ■ • ■ CL) 

M—>-oo M — 
m= 1 

= J A( qi . . . q mUm )Q(q, . . . q n di m )dq" dim = < A > . (9.105) 


9.3.7 Metropolis Algorithm 

The algorithm by Metropolis [111] can be used to select the necessary configurations. 
Starting from an initial configuration qo = (q'L ''' Tw) a c h a ' n °f configurations is 
generated. Each configuration depends only on its predecessor, hence the configura¬ 
tions form a Markov chain. 

The transition probabilities 

W^j = P(q, q/) (9.106) 

are chosen to fulfill the condition of detailed balance (Fig. 9.8) 
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Fig. 9.8 Principle of • 

detailed balance 


W 



This is a sufficient condition that the configurations are generated with probabilities 
given by their Boltzmann factors. This can be seen from consideration of an ensemble 
of such Markov chains: Let N„( q,) denote the number of chains which are in the 
configuration q, after n steps. The changes during the following step are 

AN (q,-) = N n+ 1 (q,) - iV„(q ; ) = Y N n (q i )W j ^ i - (9.108) 

qjeconf. 


In equilibrium 


N eq ( q,-) = N 0 e(qd 


(9.109) 


and the changes (9.108) vanish: 

AN (q,-) = N 0 Y 0(q- QiqdW^j 

9, 

= No Y g(qj)Wj^i - £>(q,) 

% 

= 0. (9.110) 

A solution of 

AN (q,) = Y Nn^Wj^i - N n (qi)Wi^j = 0 (9.111) 

q j€conf. 


Wi 


J^‘ 


g(q/) 

e(.qd. 


corresponds to a zero eigenvalue of the system of equations 


YN{qj)Wj^ - N (q,) Y W^j = XN (q,). 


(9.112) 
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One solution of this eigenvalue equation is given by 


N eq ((lj) _ g(q 7 ) 
Negiqd iXq,)' 


(9.113) 


However, there may be other solutions. For instance if not all configurations are 
connected by possible transitions and some isolated configurations are occupied 
initially. 

Metropolis Algorithm 

This famous algorithm consists of the following steps: 

(a) choose a new configuration randomly (trial step) with probability 


T(q, -* q trial) = T(<lt>ial -> q,) 


(b) calculate 

^ _ Qifitrial) 

iXq,) 

(c) if R > 1 the trial step is accepted q, +1 = q 

(d) if R < 1 the trial step is accepted only with probability R. choose a random 
number £ e [0, 1] and the next configuration according to 


q;+i = 


q trial if f ' R 

q/ if Z>R- 


The transition probability is the product 




-/ — f 


(9.114) 


of the probability 7)^ ; to select i -> j as a trial step and the probability A^j to accept 
the trial step. Now we have 


for R> 1 -> Aj->j = 1, Aj^i = R 1 
for R < 1 -> Aj^j — R , Aj^i = 1 

Since = 7) in both cases 

jW _ _ 

N,y/ (q ) 11/ >; O ((] ; ) 


(9.115) 


(9.116) 
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The size of the trial steps has to be adjusted to produce a reasonable acceptance 
ratio of 



(9.117) 


Multiple Walkers 

To scan the relevant configurations more completely and reduce correlation between 
the samples, usually a large number of “walkers” is used (e.g. several hundred) which, 
starting from different initial conditions, represent independent Markov chains. This 
also offers a simple possibility for parallelization. 


Problems 

Problem 9.1 Central Limit Theorem 

This computer experiment draws a histogram for the random variable r, which is 
calculated from N random numbers 



(9.118) 


Vn 


The £,■ are random numbers with zero mean and unit variance and can be chosen as 

• £; = ±1 (coin tossing) 

• Gaussian random numbers 

Investigate how a Gaussian distribution is approached for large N. 

Problem 9.2 Nonlinear Optimization 

MC methods can be used for nonlinear optimization (Traveling salesman problem, 
structure optimization etc.) [112]. Consider an energy function depending on many 
coordinates 


E{qi,q 2 - ■ ■ q N ). 


(9.119) 


Introduce a fictitious temperature T and generate configurations with probabilities 


1 

P(q l • • • <7tv) = -e 


-E(q v -q N )/T 


(9.120) 


Slow cooling drives the system into a local minimum. By repeated heating and 
cooling other local minima can be reached (simulated annealing) 
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In this computer experiment we try to find the shortest path which visits each of 
N up to 50 given points. The fictitious Boltzmann factor for a path with total length 
L is 

P(L) = e~ L/T . (9.121) 

Starting from an initial path S — (i\, h, • ■ ■ tv) n < 5 and p are chosen randomly and 
a new path S' = (ii, ■ ■ ■ i p -\, i p +„, ■ ■ ■ i p , i p +, ,+i, • • • iw) is generated by reverting the 
sub-path 

Ip ' * * lp+n ^ lp+n ' ' ' Ip • 

Start at high temperature T > L and cool down slowly. 



Chapter 10 

Eigenvalue Problems 


Eigenvalue problems are omnipresent in physics. Important examples are the time 
independent Schrodinger equation in a finite orthogonal basis (Chap. 10) 

M 

Y < <Pr\H\0j > Cj = ECf (10.1) 

j= 1 

or the harmonic motion of a molecule around its equilibrium structure (Sect. 15.4.1) 



( 10 . 2 ) 


Most important are ordinary eigenvalue problems , 1 which involve the solution of 
a homogeneous system of linear equations 



(10.3) 


with a Hermitian (or symmetric, if real) matrix [113] 



(10.4) 


The couple (A, Jt) consisting of an eigenvector x and the corresponding eigenvalue 
A is called an eigenpair. 


1 We do not consider general eigenvalue problems here. 
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Matrices of small dimension can be diagonalized directly by determining the 
roots of the characteristic polynomial and solving a homogeneous system of linear 
equations. The Jacobi method uses successive rotations to diagonalize a matrix with 
a unitary transformation. A very popular method for not too large symmetric matrices 
reduces the matrix to tridiagonal form which can be diagonalized efficiently with the 
QL algorithm. Some special tridiagonal matrices can be diagonalized analytically. 
Special algorithms are available for matrices of very large dimension, for instance 
the famous Lanczos method. 


10.1 Direct Solution 

For matrices of very small dimension (2, 3) the determinant 

det \ajj - XSij | = 0 (10.5) 

can be written explicitly as a polynomial of A. The roots of this polynomial are the 
eigenvalues. The eigenvectors are given by the system of equations 

y.(a» - AfiyOn, = 0. (10.6) 

j 


10.2 Jacobi Method 


Any symmetric 2x2 matrix 

A= ( a n 

\ a 12 «22 / 


(10.7) 


can be diagonalized by a rotation of the coordinate system. Rotation by the angle p 
corresponds to an orthogonal transformation with the rotation matrix 


( cos p — sin p 
sin p cos p 


( 10 . 8 ) 


In the following we use the abbreviations 


c = cos p, s = sin p, t — tan p 


(10.9) 
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The transformed matrix is 


RAR- l = ( C - S )( anai2 )( C S ) 

\S C ) \a\2 a 2 2 ) \-sc ) 

( c 2 a n + s 2 a 22 - 2csan cs(a u - a 2 2 ) + (c 2 - J 2 )«i 2 \ 
cs(a u - 022 ) + (c 2 - s 2 )a n s 2 a n + c 2 a 22 + 2csa 12 )' 

It is diagonal if 


0 = c,s(«i 1 - a 22 ) + (c 2 - ^ 2 )fli 2 = flH a ~ sin(2<^) + <212 cos(2<^) (10.11) 

or 

2(719 

tan(2y>) = -—. (10.12) 

a 22 ~ All 

Calculation of ip is not necessary since only its cosine and sine appear in (10.10). 
From [113] 


1 -t l c 2 -s 2 «22 — «n 

= cot(2 ip) 


t 2 cs 

we see that t is a root of 
2 a 22 — an 


2a 


12 


+ 


hence 


-t- 1 = 0 


ai2 


(10.13) 


(10.14) 


t = _^an ± 


2 a 


12 


/ a 2 2 ~ au Y _ 

\ 2an ) 




:. (10.15) 


( 322 —< 311 

2fli2 


For reasons of convergence [113] the solution with smaller magnitude is chosen 
which can be written as 



(10.16) 
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Again for reasons of convergence the smaller solution is preferred and therefore 
we take 


1 


a/I + t 2 Vl+t 2 ' 

The diagonal elements of the transformed matrix are 
«ii = c 2 an + s 2 a 22 — 2csan 


(10.17) 


(10.18) 


«22 — ^ 2 flll + C 2 a 22 + 2c5fli 2 . 


The trace of the matrix is invariant 


(10.19) 


fl 11 + fl22 = A11 + «22 


( 10 . 20 ) 


whereas the difference of the diagonal elements is 


2 , 1 —1~ ant 

flu - a 2 2 = (cT ~ s )(an - a 22 ) - 4csa n = ——r (an - a 22 ) - 4- 


l+t 2 


1 + t 2 


1 — f 2 —2t 2 a\ 2 t 

= (fill — fl 22 ) + (—■fl !2 —) z—— 4 —— 2 = (ail — « 22 ) — 2tfli2 


1 +t 2 


1 + t 2 


( 10 . 21 ) 


and the transformed matrix has the simple form 

V ( 10 . 22 ) 

\ a 22 + ant) 

For larger dimension N > 2 the Jacobi method uses the following algorithm: 

(1) look for the dominant non-diagonal element max | a, ; - 

(2) Perform a rotation in the (z'/j-plane to cancel the element a, ; of the transformed 
matrix A = R tij> ■ A ■ R (j2> ' 1 . The corresponding rotation matrix has the form 


/ ! 


\ 


R di) = 


c s 

—s c 


V 



(10.23) 


(3) repeat (1-2) until convergence (if possible). 
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The sequence of Jacobi rotations gives the over all transformation 


RAR~ l = ■R 2 R\AK[ l R~ l ■ ■■ 



(10.24) 


Hence 


AR~ l = R~ l 



(10.25) 


and the column vectors of R 1 = (Vi, Vt • • • v,v) are the eigenvectors of A: 


A (vi, \2 ■ ■ ■ vjv) = (AiVi, A 2 V 2 , • • • XnVn) ■ 


(10.26) 


10.3 Tridiagonal Matrices 

A tridiagonal matrix has nonzero elements only in the main diagonal and the first 
diagonal above and below. Many algorithms simplify significantly when applied to 
tridiagonal matrices. 

10.3.1 Characteristic Polynomial of a Tridiagonal Matrix 

The characteristic polynomial of a tridiagonal matrix 


Pa( A) = det 


an — A a\2 

<421 <422 — A 


(10.27) 


On-in 
(Inn-i a NN — A 


can be calculated recursively: 


Po = 1 


Pi (A) = a n - A 


PlW = («22 — A)Pi(A) — fll2«21 


Pn( A) = (fiNN — A)Pat-i(A) — aN,N-\ON-\,NPN-2{A). 


(10.28) 
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10.3.2 Special Tridiagonal Matrices 

Certain classes of tridiagonal matrices can be diagonalized exactly [114-116]. 


10.3.2.1 Discretized Second Derivatives 


Discretization of a second derivative involves, under Dirichlet boundary conditions 
f(x o) —/(xn+ i) = 0, the differentiation matrix (Sect. 20.2) 


/- 2 1 

1-2 1 


M = 


1 -2 1 
1 - 2 / 


(10.29) 


Its eigenvectors have the form 



//l\ 


/ sin k \ 


/ = 

fn 

= 

sin (nk) 

(10.30) 




^sin (Nk) j 



This can be seen by inserting (10.30) into the n-th line of the eigenvalue (10.31) 
Mf = \f (10.31) 


(M/)„ = (sin ((« — 1 )k) + sin ((« + l)k) — 2 sin(nk)) 

= 2 sin (nk) (cos (k) - 1) = A (f) n (10.32) 

with the eigenvalue 


A = 2 (cos k — \ ) = —4 sin 2 



The first line of the eigenvalue (10.31) reads 


(10.33) 


(M/)j = (—2 sin(Ar) + sin(2£)) 
= 2 sin (k) (cos (k) — 1) = A if). 


(10.34) 
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and from the last line we have 

(Mf) N = (—2 sin(M') + sin([iV — l]/c)) 
= A(f)w = 2(cos(£) — 1) sin(M’) 

which holds if 

sin((iV — 1 )k) — 2 sin (Nk) cos (k). 


This simplifies to 


sin(M:) cos(fe) — cos (Nk) sin(£) = 2 sin(Nk) cos (k) 
sin(Nk ) cos (k) + cos (Nk) sin(&) = 0 
sin((iV + 1 )k) — 0. 

Hence the possible values of k are 

k = ——-/ with l = 1,2, •••N 

(N+ 1) 

and the eigenvectors are explicitly (Fig. 10.1) 


/ = 


/ sin (ivTlO \ 
sin G£j/ n) 
Vsin (jfclN)/ 


(10.35) 


(10.36) 


(10.37) 


(10.38) 


(10.39) 


For Neumann boundary conditions %.(x\) — ^(xn) = 0 the matrix is slightly 
different (Sect. 20.2) 


Fig. 10.1 (Lowest 
eigenvector) Top for fixed 
boundaries/„ = sin(n£) 
which is zero at the 
additional points xo, xp+\. 
Bottom for open boundaries 
/„ = cos ((n — 1 )k) with 
horizontal tangent at x\, x;v 
due to the boundary 
conditions 

fl =fo,fN-l =fh +1 




x 
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/ —2 2 
1-2 1 


M 


1 -2 1 


- 2 / 


(10.40) 


Its eigenvalues are also given by the expression (10.33). To obtain the eigenvectors, 
we try a more general ansatz with a phase shift 


/ sin 0i \ 


f = 


sin(<£i + (n — 1)£) 


Vsin^! + (N- \ )k) / 


(10.41) 


Obviously 


sin(<Ji + (n — l)k — k) + sin(<J>i + (n — \)k + k) — 2 sin(<Ji + (n — 1 )k) 

— 2 (cos k — 1) sin(<? 1 + (n — 1)A')- (10.42) 


The first and last lines of the eigenvalue equation give 

0 = — 2 sin(<£i) + 2 sin(<£i + k) — 2(cosA: — 1) sin(0i) 

= 2cos<£i sinA: (10.43) 


and 

0 = -2 sin(0i + (TV - 1)£) + 2 sin^ + (TV — 1 )k - k) 

— 2(cosA: — 1) sin( Cf*! + (N — l)k) = 2cos(cf> 1 + (TV — 1)£) sin A: (10.44) 

which is solved by 

0i = - k=-^—l, l=\,2...N (10.45) 

2 TV- 1 

hence finally the eigenvector is (Fig. 10.1) 


/ = 


/ 1 \ 

:os (^T 7r/ ) 


V (-1)' 


(10.46) 
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Even simpler is the case of the corresponding cyclic tridiagonal matrix 


-2 

1 -2 1 


M = 


1 \ 


1 -2 1 

1 1-2 / 


(10.47) 


which has eigenvectors 


/e*\ 



(10.48) 


and eigenvalues 

A = -2 + e~ ik + e ik = 2 (cos (k) - 1) = -4 sin 2 f (10.49) 

where the possible k — values again follow from the first and last line 

—2e ik + e i2k + e* = (-2 + t~ ik + e ik ) e ik (10.50) 

e ik + e iw ~ 1,k - 2e im = (-2 + e~ ik + e ik ) e* (10.51) 

which both lead to 

e m = 1 (10.52) 

2-7T 

k=—l, I = 0, l, ■ ■ ■ N — l. (10.53) 

N 

10.3.2.2 Discretized First Derivatives 

Using symmetric differences to discretize a first derivative in one dimension leads 
to the matrix 2 


2 This matrix is skew symmetric, hence i T is Hermitian and has real eigenvalues iA. 
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/ 


-1 


D = 


\ 


\ 

-1 1 
-1 


(10.54) 


The characteristic polynomial of the Hermitian matrix i D is given by the recursion 
Sect. 10.3.1 


Po = 1 
Pi = - A 


P N = -\P N - l -P N -2 (10.55) 

which after the substitution x = —A/2 is exactly the recursion for the Chebyshev 
polynomial of the second kind Un(x). Hence the eigenvalues of D are given by the 
roots Xk of U N (x) as 


A d — 2i %k = 2icos 


(—) ‘ = 
VV+1/ 


= 1,2 ...N. 


The eigenvalues of the corresponding cyclic tridiagonal matrix 

-1\ 


D = 


/ 1 

-1 1 


V 1 


-1 1 
-1 / 


are easy to find. Inserting the ansatz for the eigenvector 
exp i k 

. exp L Nk, 

we find the eigenvalues 


(10.56) 


(10.57) 


(10.58) 


e i(m+l)t- _ e i(m— l)k _ 


(10.59) 


A = 2i sin k 


(10.60) 
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and from the first and last equation 


(10.61) 


s lk = gHA'+D* 


(10.62) 


the possible A:-values 


k=—l, 1 = 0, 1. 

N 


(10.63) 


10.4 Reduction to a Tridiagonal Matrix 

Eigenproblem algorithms work especially efficient if the matrix is first transformed 
to tridiagonal form (for real symmetric matrices, upper Hessian form for real non- 
symmetric matrices) which can be achieved by a series of Householder transforma¬ 
tions (5.56) 


A' = PAP with P = P T = 1-2—-. 

\u \ 2 


(10.64) 


The following orthogonal transformation P i brings the first row and column to tridi¬ 
agonal form. We divide the matrix A according to 



(10.65) 


with the (N — 1)- dimensional vector 


a = 


( 



Now let 


u = 



V a lN ) 


0 


( 10 . 66 ) 


Vo/ 
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12 


Then 

|m| 2 = \a\ 2 + A 2 + 2A a 
and 

= \a\ 2 + \a l2 . 


■'(a)" 


The first row of A is transformed by multiplication with Pi according to 

Ictl 2 + A«i 


P i 


(«)=(«) 


ni 


0 


\a\ 2 + A 2 + 2A«i2 
The elements number 3 ... N are eliminated if we choose 3 
A = ±|a| 
because then 

|a| 2 + Aai2 . |a| 2 ± \a\ai2 


Ae< 2 > 


a| 2 + A 2 + 2Afli2 |a| 2 + |a| 2 ± 2|a|ai2 


= 1 


and 




Ae (2) = 


/ an \ 
T\a 
0 

V 0 ) 


Finally we have 


A a) = P\APi = 


( an 

(2) 
a 12 

0 • 

■ 0 \ 

J2) 

J2.) 

J2) 

J2) 

a n 

a 22 

a 23 ' 

■ a 2N 

0 

a (2) 

“23 


(2) 

a 3N 

l o 

(2) 

a 2N 

(2) 

a 3N ' 

. a (2) y 
“AW / 


as desired. 


3 To avoid numerical extinction we choose the sign to be that of a\ 2 - 


(10.67) 


( 10 . 68 ) 


(10.69) 


(10.70) 


(10.71) 


(10.72) 


(10.73) 
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For the next step we choose 


2 (2) 

i 22 


a (2) 
\ a 2N / 


O' 

u = | 0 | ± |a|e (3) 
a, 


(10.74) 


to eliminate the elements a 24 ■ .. « 2 ,v- Note that P 2 does not change the first row and 
column of A {1) and therefore 


A (3) = P 2 A a) P 2 = 


/ flu fl® 0 . 0 \ 

fll2 a 22 «23 0 ' ' ' 0 

‘ (3) 

3 JV 


0 fl^flg 


a. 


0 


V 0 0 


(3) 

2 3N 


(3) 

2 NN 


(10.75) 


After N — 1 transformations finally a tridiagonal matrix is obtained. 

10.5 The Power Iteration Method 

A real symmetric N x N matrix with (orthonormal) eigenvectors and eigenvalues 4 
Auj = AjUj (10.76) 

can be expanded as 

A = ^iiiXiuJ. (10.77) 


The sequence of powers 
A" = 


(10.78) 


converges to 


A" —> u max A" u T 

“max ' '■ m ax max 


4 We do not consider degenerate eigenvalues explicitly here. 
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where 5 


\^mwc\ = max. 


(10.79) 


Hence for any initial vector v\ (which is arbitrary but not perpendicular to u max ) the 
sequence 

v n+i =Av n (10.80) 


converges to a multiple of u max . To obtain all eigenvectors simultaneously, we could 
use a set of independent start vectors, e.g. the IV unit vectors and iterate simultaneously 
for all of them 




vf'') = («i. ■■■e N ) = 






(10.81) 

(10.82) 


Most probably, all column vectors of A then converge to multiples of the same 
eigenvector. To assure linear independence, an orthogonalization step has to follow 
each iteration. This can be done (QR decomposition, Sect. 5.2) by decomposing the 
matrix into the product of an upper triangular 6 matrix R and an orthogonal matrix 
Q T = Q~ x (Sect. 5.2) 


A = QR. 


(10.83) 


For symmetric tridiagonal matrices this factorization can be efficiently realized by 
multiplication with a sequence of Givens rotation matrices which eliminate the off- 
diagonal elements in the lower part one by one 7 


n=R (N ~ 1 ’ N) o(2,31 £(1,2) 

&N -1 ‘ ' ' OL2 ai 

beginning with 


(10.84) 


/ 


R^A = 


c s 
-s c 


\ / ^11 ^12 

a \2 CI22 ail 


1 / 


d/V-2,JV-l ClN-l,N-l flw-l.v 
&N-1,N a N,N J 


5 For simplicity we do not consider eigenvalues which are different but have the same absolute value. 
6 The equivalent QL method uses a lower triangular matrix. 

7 This is quite different from the Jacobi method since it is not an orthogonal transformation. 
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/ can + sci\2 cai2+sa22 sa23 
0 -san + ca 22 ca23 


<734 


a 33 a 3A 


un-\,n on,n / 


(10.85) 


where the rotation angle a\ has to be chosen such that 


s a 17 

tan 04 = - = —. ( 10 . 86 ) 

c an 

Finally, this leads to a method known as orthogonal simultaneous power iteration 
W (1) = A = (10.87) 


W (n+1) = AQ (n) (10.88) 

Q(n+\) R (n+Y) _ w (n+ 1 )_ (10.89) 

This method calculates a sequence of orthogonal matrices (Q [n) which converge to a 
set of independent eigenvectors. Moreover, from (10.88) and (10.89) 

A = W (n+1) Q (n)T = g ( " + i ) 7? ( " + i ) <2 ( " )7 ' (10.90) 

and therefore powers of A are given by 

A" = (Q M R M Q (n ~ 1)r ) (g ( " _1) ,/?(«-»g ( "“ 2)7 ’) ... (qWRW Q a)T ) (Q (1) R (1) ) 

= Q (n) R in) R 0, - l) (10.91) 

The product of two upper triangular matrices is upper triangular again which can be 
seen from 

{R (m) R ( "\ k = ^ R^Rjk = 0 if i > k. (10.92) 

i<j<k 

Therefore the QR decomposition of A" is 

A" = Q {n) R [n) (10.93) 


with 


R (n> = R in) ...R (t) . 


(10.94) 
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To obtain other than the dominant eigenvalues, the inverse power iteration method 
with shift is useful. Consider the matrix 


A = (A - ay 1 


(10.95) 


where [i is not an eigenvalue A,of A. Obviously it has the same eigenvectors as A 
and eigenvalues given by 


Auj = \jiij = - Uj. 

A i - a 


(10.96) 


Hence, if a is close to A, , the power iteration method will converge to a multiple of m, . 
For practical calculations, an equivalent formulation of the power iteration method 
is used which is known as the QR (or QL) method. 

10.6 The QR Algorithm 

The QR algorithm [117] is an iterative algorithm. It uses a series of orthogonal trans¬ 
formations which conserve the eigenvalues. Starting from the decomposition of A 


A = Q\R\ 


(10.97) 


^2 = ^101 = 0 ^ 0 ! 


(10.98) 


we iterate 


K = Q„R„ 


(10.99) 


An +1 — RnQn — Q n A n Q n 


( 10 . 100 ) 


From (10.99) and (10.100) 


Qn+\R n +l — RnQn 


and the n-th power of A is 


A n = AA .. .A = 2itfi2ifli • ■ ■ Q\R\ = QAQ 2 R 2 ■ ■ ■ 22^2)^! 

= 0102 ( 03^3 • • ■ 03 ^ 3 )^ 2^1 ‘ = Q n Rn 


( 10 . 101 ) 
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But since QR decomposition is unique, comparison of (10.93) and (10.101) shows 
Q„ = Q (n) Rn = R (n) (10.103) 

i.e. the column vectors of Q n converge to a set of eigenvectors and the transformed 
matrix 


A n+ 1 = Q T n A„Qn = QnQn-\A n -\Qn-\Q n = ■■'= Q„AQ„ (10.104) 

converges to a diagonal matrix. Now consider the inverse power 

A~ n =R~ l Q T n . (10.105) 

The inverse of a symmetric matrix is also symmetric and 

A~ n = Q n (V ) 1 (10.106) 

shows, that the QR algorithm uses the same orthogonal transformations as ordinary 
and also inverse power iteration. The inverse of an upper triangular matrix is also 
upper triangular but the transpose is lower triangular. Therefore we modify (10.106) 
by multiplying with a permutation matrix 



from the right side, which reverses the order of the columns and 
A~"P = Q n PP (V ) 7 P = QR 


(10.107) 


(10.108) 


is the QR decomposition of A~"P. This shows the close relationship between the QR 
algorithm 8 and the inverse power iteration method. 

To improve convergence, a shift a is introduced and the QR factorization applied 
to A„ — a. The modified iteration then reads 

A n - a = Q„R„ (10.109) 


A „+1 — R„Q n + cr — QT(A„ — a)Q n + a — QlA n Q„ 


( 10 . 110 ) 


Or the equivalent QL algorithm. 



230 


10 Eigenvalue Problems 


Symmetry and tridiagonal form are conserved by this algorithm. The simplest 
choice for the shift is to take a diagonal element a m = a m m = (A„) mm corresponding 
to the Rayleigh quotient method. An even more robust and very popular choice [113] 
is Wilkinson’s shift 


<y m = a m ,m + S - sign(<5),y 5 2 + a ^ m _j S = l,m ^ (10.111) 

which is that eigenvalue of the matrix I U '" 1,m J which is closer to a n n . 

\ &m,m J 

The calculation starts with ctn and iterates until the off-diagonal element 
becomes sufficiently small. 9 Then the transformed matrix has the form 


/flu fli2 \ 

Al2 A22 


V 


On-2,N- i 

<*N-2,N-l On-I.N-I 0 

0 am / 


( 10 . 112 ) 


Now the last column and row can be discarded (deflation method) and the next 
iteration performed with the shift ctjv-i on a tridiagonal matrix of dimension N — 1. 
This procedure has to be repeated N times to obtain all eigenvalues. Convergence is 
usually cubic (or at least quadratic if there are degenerate eigenvalues). 


10.7 Hermitian Matrices 

In quantum mechanics often Hermitian matrices have to be diagonalized (which 
have real valued eigenvalues). To avoid complex arithmetics, an Hermitian eigen- 
problem can be replaced by a symmetric real valued problem of double dimension 
by introducing 


B — 91(A) C = £s(A) x = u + iv 


(10.113) 


where, for Hermitian A 

A = B + iC = A H — B T — iC T 


(10.114) 


hence 


B — B t C= -C t 


(10.115) 


9 For the QL method, it is numerically more efficient to start at the upper left corner of the matrix. 
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and the eigenvalue problem can be rewritten as 


0 = (Ax — Ax) = (Bu — Cv — Am) + i(Bv + Cu — Ad) 


(10.116) 


or finally in the real symmetric form 



(10.117) 


Each eigenvalue of the N-dimensional Hermitian problem corresponds to two eigen¬ 
vectors of the 2N-dimensional problem since for any solution of (10. 1 17) 



(10.118) 


provides a different solution, while the complex vectors u + id and i (u + iw) = 
—v + iu only differ by a phase factor. 


10.8 Large Matrices 

Many problems in computational physics involve very large matrices, for which 
standard methods are not applicable. It might be even difficult or impossible to keep 
the full matrix in memory. Here methods are used which only involve the product of 
the matrix with a vector which can be computed on the fly. Krylov methods are very 
similar to power iteration but diagonalize only the projection of the matrix onto a 
Krylov space of much smaller dimension n <<£ N which is constructed by multiplying 
a normalized start vector q x repeatedly with A 


K n (A, q x ) = span {q u Aq 1 ,A 2 q l - A" l q x }. 


(10.119) 


We use the Arnoldi method (Sect. 5.6.5) to construct an orthonormalized basis of this 
space. For a symmetric matrix this simplifies to a three-term recursion also known 
as symmetric Lanczos algorithm [118]. Applying the Arnoldi method 


hj.n = ( qjAq n ) j < n 


( 10 . 120 ) 


n 



( 10 . 121 ) 


hn+l,n — l?«+ll (7 h+1 — , 

h, 


hi+l,n 


( 10 . 122 ) 
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to a symmetric matrix A, we find 

h 2 n+i,„ = qWq„ ~ 2 ^(ql A qj)(qjMn) + ^,(qj A q„)(qJ> A q„)tjf 


= qiA 


hn+l,nq n +l A z M, 


^ hjn — hn+l,nhn,. 


n +1 


hence 

1- 

Furthermore, 

h„- 2 ,n = ql- 2 A <ln = <lI,- 2 A 


hn.n—1 


n— 1 

Mn-l -^bjn-lQj 
1=1 


1 


hn.n -1 

1 

K.n-\ 


~ ^ hj n -lh],„-2 ~ h n -l,n-lhn-2,n-l + ql-l A I K-l,n-2q n -\ + ^ fyn-2?. 


1=1 
n —2 


1=1 


hjn— lhj,n—2 hn—l,n—lhn—2,n—l "b l,/i— l^bi—l,n—2 3“ ^ hjn — ] hjn —2 

1=1 1=1 


and similar for s > 2 


= ?„_ S A?„ = ?„_ S A 


hn.n—\ 


n— 1 

A?,,.! *£/l>,-l?j 
1=1 


1 


hn.n—l 

1 

hn.n-l 

1 

hn.n— 1 


-'^_,hj„-\hj. n - s — hn—l.n— 1 h n —s,n— 1 + qJ,-l A ( h n - s+ \ ,n-sQ n -s +1 + Z h]„- 
1=1 \ 1=1 
«—2 n—s 

hjn— 1 hj.n— s h n —\ t n—\h n — Ss n—\ "b h n — s+l,n— shn— 1,«—s-t-1 "b ^ hj n — | hjn 
1=1 1=1 


^ ' hj.n—lhj.n—s h n —\,n—\hn—s,n—\ "b h n — S n—s+lh n —s+l,n—l 
j=n-s +1 


lln—s.n — 


hn.n—] 


n —2 


^ hj.n— i hj.n—s h n _\ n \h n - sn -\ 
j=n-s+2 


Starting from (10.125) for j = 2 we increment s repeatedly and find 


(10.123) 


(10.124) 


= o 

(10.125) 



(10.126) 


(10.127) 


hn—2,n — 3 ,h — ... Ill n — 0 


(10.128) 
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since (10.127) only involves smaller values of s, for which (10.128) already has been 
shown. The Arnoldi decomposition produces an upper Hessenberg matrix Sect. 5.6.5, 


U n = (q x ,...q n ) 


(10.129) 


AU n = U n+1 H = (U n , q ll+l ) ( Hn r ) = U„H n + h n+l q n+1 e T n (10.130) 
which for symmetric A becomes tridiagonal 


H = 

l hn hn ■ 
hn hi2 ■ 

^32 

• h\n \ 

• ^2 n 


a\ b i 
b\ a 2 b2 

\ 



• hnn 

hfi+\,n ) 


bn—2 An— 

bn— 

\ 

bn— 1 

b n / 



(10.131) 


with a symmetric tridiagonal matrix T, which is the desired projection of A into the 
Krylov space K n 


Un A U n = U T n U n+l H= Ul (U n , q n+ \) 

= (e " 0 , ('i) = ^ <10132) 

For an eigenpair (A, v) of T 

A(U n v) = U n+l Hv = (U n ,q n+1 ) v 

= ( U„T + b n q n+l e T „) v = X(U n v) + b n q n+l ej t v. (10.133) 


Hence, an approximate eigenpair of A is given by the Ritz pair (A, U n v) and the error 
can be estimated from the residual norm 


I (A — A) (7n it | 

1*1 


\b„ 


\elv\ 


(10.134) 


Due to numerical errors, orthogonality of the Lanczos vectors q n can get lost and 
reorthogonalization is necessary [119, 120]. If this takes to much time or if memory 
limits do not allow to store enough Lanczos vectors, the procedure has to be restarted 
with a new initial vector which is usually taken as a linear combination of selected 
eigenvectors which have already been found [121, 122]. Furthermore, special care 
has to be taken to determine possible degeneracies of the eigenvalues. 
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10.9 Non-symmetric Matrices 

Eigenvalue problems with non-symmetric matrices are more complicated. Left and 
right eigenvectors have to be distinguished and the eigenvalues can be complex valued 
even if the matrix is real. The QR method [117] is applicable also to a non-symmetric 
matrix but very expensive unless the matrix is first brought to upper triangular (instead 
of tridiagonal) form, which can be achieved by a series of similarity transformations 
with Householder reflections (5.2.2). The implicit QR method with double shift 
avoids complex arithmetics by treating pairs of complex conjugated eigenvalues 
simultaneously. For very large matrices the Arnoldi method brings a non-symmetric 
matrix to upper Hessenberg form, which provides the projection onto the Krylov 
space as an upper triangular matrix. 


Problems 

Problem 10.1 Computer Experiment: Disorder in a Tight-Binding Model 

We consider a two-dimensional lattice of interacting particles. Pairs of nearest neigh¬ 
bors have an interaction V and the diagonal energies are chosen from a Gaussian 
distribution 


P(E) = — l —e- E2/2A2 . (10.135) 

As/2/ 

The wave function of the system is given by a linear combination 

= (10.136) 

y 


where on each particle (i,j) one basis function ipy is located. The nonzero elements 
of the interaction matrix are given by 


H(ij\ij)=E ij (10.137) 

H(ij\i ± 1,;) = H(ij\iJ ± 1) = V. (10.138) 

The Matrix H is numerically diagonalized and the amplitudes Cy of the lowest 
state are shown as circles located at the grid points. As a measure of the degree of 
localization the quantity 

2>y| 4 (10.139) 

ij 


is evaluated. Explore the influence of coupling V and disorder A. 



Chapter 11 

Data Fitting 


Often a set of data points has to be fitted by a continuous function, either to obtain 
approximate function values in between the data points or to describe a functional 
relationship between two or more variables by a smooth curve, i.e. to fit a certain 
model to the data. If uncertainties of the data are negligibly small, an exact fit is 
possible, for instance with polynomials, spline functions or trigonometric functions 
(Chap. 2). If the uncertainties are considerable, a curve has to be constructed that 
fits the data points approximately. Consider a two-dimensional data set 

(Xi, yd i=l...m (11.1) 

and a model function 

f(x, a\ ... a n ) m > n (11.2) 

which depends on the variable x and n < m additional parameters aj. The errors of 
the fitting procedure are given by the residuals 

r, = y t -f(xt, a\ ... a„). (11-3) 

The parameters aj have to be determined such, that the overall error is minimized, 
which in most practical cases is measured by the mean square difference 1 

1 m 

S sd (a\.. ,a n ) — — TV 2 . (11.4) 

m z —' 

i -1 

The optimal parameters are determined by solving the system of normal equations. 
If the model function depends linearly on the parameters, orthogonalization offers a 
numerically more stable method. The dimensionality of a data matrix can be reduced 


1 Minimization of the sum of absolute errors Y \ r; is much more complicated. 
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with the help of singular value decomposition, which allows to approximate a matrix 
by another matrix of lower rank and is also useful for linear regression, especially 
if the columns of the data matrix are linearly dependent. 


11.1 Least Square Fit 


A (local) minimum of (11.4) corresponds to a stationary point with zero gradient. 
For n model parameters there are n, generally nonlinear, equations which have to be 
solved [123]. From the general condition 

^=0 j=l...n (11.5) 

daj 

we find 


z 


df(xj, a\ ... a„) 
d aj 


= 0 


( 11 . 6 ) 


which can be solved with the methods discussed in Chap. 6. For instance, the 
Newton-Raphson method starts from a suitable initial guess of parameters 

(«?■■■ a° n ) (11.7) 


and tries to improve the fit iteratively by making small changes to the parameters 
a] +1 = a] + Aa], (11.8) 

The changes A a] are determined approximately by expanding the model function 


f(xi, a] +1 ... a; 


"' +1 ) —f(xi, a] ... a' n ) + 

j= i 


df(xj, a\... a s n ) 


daj 


Aa] + .. 


(11.9) 


to approximate the new residuals 

,V+I = , _ V 9f(Xj, a\... a s m ) 

' ' ^ daj 

7=1 

and the derivatives 

dr] _ df(x h a\...a] n ) 
daj daj 


Aa] 


( 11 . 10 ) 


( 11 . 11 ) 
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Equation (11.6) now becomes 


z 




Qfixj) 

dak 


( 11 . 12 ) 


which is a system of n (usually overdetermined) linear equations for the Aaj, the 
so-called normal equations: 


df (xj) df (xj) . s _ 'v -1 j df (xj) 
daj dak 7 , 1 dak 

i= 1 7=1 J i=l 


(11.13) 


With the definition 


4 _ 1 "ST d f{Xi) df(xi) 

k ' m dak da: 

1=1 J 

(11.14) 

1 df (xj) 

b k = > y, 

m oak 

i=i 

(11.15) 

the normal equations can be written as 


n 

AkjAaj = bk- 

7=i 

(11.16) 

11.1.1 Linear Least Square Fit 


Especially important are model functions which depend linearly 
(Fig. 11.1 shows an example which is discussed in problem 11.1) 

on all parameters 

n 

fix , Oi ... a n ) = ^ a/jix). 

7=1 

(11.17) 

The derivatives are 


df(xi) 

Ba, =/l(Xl) 

(11.18) 


and the minimum of ( 11.4) is given by the solution of the normal equations 
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Fig. 11.1 (Least square fit) 
The polynomial 
C(T) = aT + bT 3 (full 
curve) is fitted to a set of data 
points which are distributed 
randomly around the “exact” 
values C(T) = agT + bgT 3 
(.dashed curve). For more 
details see problem 11.1 



T 


1 n m 1 m 

— V y'/k(.Xi)fj(Xi)a j = — T.ydkixj) 

m z — 4 z — 4 m z ' 

7=1 i= 1 i= 1 

which for a linear fit problem become 

n 

T A ^j = 

j =i 


with 


A kj = 


- y'jk(xi)fj(xi) 

,n ' 

i= 1 

^ m 

bk = — 'T. ydkix,). 

m z —' 

1 =! 

Example: Linear Regression 

For a linear fit function 

fix) = ciq + a\x 
the mean square difference is 


1 m 

Ssd — ^ 0; 

n? ' 


«0 — U\Xj) 


(11.19) 


( 11 . 20 ) 


( 11 . 21 ) 

( 11 . 22 ) 


(11.23) 


(11.24) 


and we have to solve the equations 
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dS sd 1 ^ 

0 = —— = > (yi -ao- a\Xi) = y - a Q - a x x 

m ‘ J 


dao m 


1= 1 


0 = 


dS. 


sd 


. m 

= 1 Zb-. 

vn *« J 


da i m A 


flo — a\Xi)Xj = xy — qqx — a\X 2 


1=1 


which can be done here with determinants 


ao 


y 

xy x 2 


1 x 


yx 2 — xxy 

X 2 — X 2 


a i 


x xy 


1 x 


xy — xy 
x 2 — X 2 


(11.25) 


(11.26) 


(11.27) 


11.1.2 Linear Least Square Fit with Orthogonalization 

With the definitions 


/«'\ 

b = 

/.Vl\ 

\a„ ) 


\y m J 


(11.28) 


and the m x n matrix 

t7l 1 • Cl\ n \ 


A = 


f\(X\) ' ' ' fn(x l) \ 

drill ' ‘ * a m n J \,/l (Xfn ) * * ' fn (x m ) J 


(11.29) 


the linear least square fit problem (11.20) can be formulated as a search for the 
minimum of 


|Ax — b| = I (Ax — b) r (Ax — b). 

In the last section we calculated the gradient 
d\Ax — b' 2 


dx 


= A r (Ax - b) + (Ax - b) r A = 2A J Ax - 2A r b 


(11.30) 


(11.31) 
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and solved the normal equations 


A t Ax = A r b. 


(11.32) 


This method can become numerically unstable. Alternatively we use orthogonaliza- 
tion of the n column vectors a/ f of A to have 


( ru n 2 ■ ■ ■ >'\n\ 
r 22 ■ ■ ■ r 2n 


A = (ai ■ ■ • a„) = (qi • • • q„) 


(11.33) 


V 


where a* and q t are now vectors of dimension m. Since the q t are orthonormal 
qf q k = S; k we have 


/qf N 


/ rn n 2 ■ 

■ Tin \ 

A = 

m ■ 

' Tin 


vq nj 


\ 

Tnn / 


The q /: can be augmented by another (in — n) vectors to provide an orthonormal basis 
of R m . These will not be needed explicitly. They are orthogonal to the first n vectors 
and hence to the column vectors of A. All vectors q* together form an orthogonal 
matrix 


0= (qi q» q»+t q m ) 


(11.35) 


and we can define the transformation of the matrix A: 


/ \ 



(11.36) 



The vector b transforms as 



qf\ K + t\ 


b b/ = 

q«7 V q l ) 


b. 


(11.37) 
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Since the norm of a vector is not changed by unitary transformations 

|b — Ax | = -J (b„ — Rx) 2 + bj 
which is minimized if 
Rx = b„. 

The error of the fit is given by 
|b — Ax | = |b,|. 

Example: Linear Regression 

Consider again the fit function 

f(x) — ao + a\x 


for the measured data (x,, y,). The fit problem is to determine 



Orthogonalization of the column vectors 


l l \ 


/Xi\ 

: 

a 2 = 


v) 


y J 


with the Schmidt method gives: 
ru = Vm 



I'12 


m 

X/ = y/mx 

i=i 


b 2 = fa - x) 


(11.38) 

(11.39) 

(11.40) 

(11.41) 

(11.42) 

(11.43) 

(11.44) 

(11.45) 

(11.46) 


(11.47) 
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r 22 


= J^(*i - x ) 2 = Vmo 


<12 = ( 



Transformation of the right hand side gives 



and we have to solve the system of linear equations 


(’“"')=( /si.)- 

V 0 Jmo ) / \ V m J 

The solution 


xy — x y 
(x — x) 2 


ao = y — xci\ 


yx 2 — x xy 
{x — x) 2 


coincides with the earlier results since 


(x — x) 2 = x 1 — x 2 . 


(11.48) 

(11.49) 

(11.50) 

(11.51) 

(11.52) 

(11.53) 

(11.54) 


11.2 Singular Value Decomposition 

Computational physics often has to deal with large amounts of data. Singular value 
decomposition is a very useful tool to reduce redundancies and to extract the most 
important information from data. It has been used for instance for image compression 
[124], it is very useful to extract the essential dynamics from molecular dynamics 
simulations [125, 126] and it is an essential tool of Bio-informatics [127]. 
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11.2.1 Full Singular Value Decomposition 

For m > n , 2 any real 3 m x n matrix A of rank r < n can be decomposed into a 
product 

A = UT,V t (11.55) 

/ an ■■■ a\ n \ 

\ a m 1 . . . 0-tnn J 

where U is a m x m orthogonal matrix, S is a m x n matrix, in which the upper part 
is a n x n diagonal matrix and V is an orthogonal n x n matrix. 

The diagonal elements s,- are called singular values. Conventionally, they are sorted 
in descending order and the last n — r of them are zero. For a square n x n matrix 
singular value decomposition (11.56) is equivalent to diagonalization 

A=USU t . (11.57) 


( U\i ... U\ m \ 

\ Mm 1 • • • M mm J 


/ Si \ 


VO ... oj 


/ v u ■ 

■ V n \ \ 

^ Vln • 

• Vnn ) 


(11.56) 


11.2.2 Reduced Singular Value Decomposition 


We write 


U = ((/„, (/„,_„) 


(11.58) 


with the m x n matrix U„ and the m x (m — n) matrix and 

M») 


(11.59) 


with the diagonal n x n matrix S. The singular value decomposition then becomes 

A = ([/„, £/,„_„) (JQ = U n SV T (11.60) 

which is known as reduced singular value decomposition. U n (usually simply denoted 
by U ) is not unitary but its column vectors, called the left singular vectors, are 
orthonormal 


2 Otherwise consider the transpose matrix. 

3 Generalization to complex matrices is straightforward. 
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m 

llj r Uj S — S r ^ s (11.61) 

i= 1 

as well as the column vectors of V which are called the right singular vectors 


n 

^ v = S rs . 

i= 1 


Hence the products 
U T n U n = V T V = E n 


(11.62) 


(11.63) 


give the n x n unit matrix. 

In principle, U and V can be obtained from diagonalization of A 1 A and ,4,4 7 , 
since 

A r A = (yZ T U T )(U£V T ) = V(S , 0) V T = VS 2 V T 

AA t = ( UZV T )(VZ T U T ) = U (S, 0 )U T = U n S 2 Ul 

However, calculation of U by diagonalization is very inefficient, since usually only 
the first n rows are needed (i.e. U„ ). To perform a reduced singular value decompo¬ 
sition, we first diagonalize 

A t A=VDV t (11.66) 

which has positive eigenvalues di > 0, sorted in descending order and obtain the 
singular values 


(11.64) 

(11.65) 


S = D ' 12 = 




Now we determine a matrix U such, that 


A = USV T 
or, since V is unitary 


(11.67) 


( 11 . 68 ) 


Y = AV = US. 


(11.69) 
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The last n — r singular values are zero if r < n. Therefore we partition the matrices 
(indices denote the number of rows) 


(Y r 0„_,.) = (Ur Q ) = (U r Sr 0) . 

We retain only the first r columns and obtain a system of equations 


/ yu ■ 

• yir \ 

( «11 • 

• Ml,. \ 

(\ 

\ 

\ ym 1 • 

■ y,nr J 

^ ^ml • 

. u mr j 

l 

SrJ 


which can be easily solved to give the first r rows of U 
/ «11 ■ •• Mlr \ 

\ M-ml • • • J 






/ yu ■ ■ ■ yin \ />v. 


\ y m \ ■ - ■ ynm / 


S r l ) 


(11.70) 


(11.71) 


(11.72) 


The remaining n — r column vectors of U have to be orthogonal to the first r columns 
but are otherwise arbitrary. They can be obtained for instance by the Gram Schmidt 
method. 

For larger matrices direct decomposition algorithms are available, for instance 
[128], which is based on a reduction to bidiagonal form and a variant of the QL 
algorithm as first introduced by Golub and Kahan [129], 


11.2.3 Low Rank Matrix Approximation 

Component-wise (11.60) reads 

r 

di j — Mi,kSk ty,k • (11.73) 

k= 1 


Approximations to A of lower rank are obtained by reducing the sum to only the 
largest singular values (the smaller singular values are replaced by zero). It can be 
shown [130] that the matrix of rank l < r 

i 

a { /j = ^ Ui'kSkVj,k 
k =1 


(11.74) 
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is the rank-/ matrix which minimizes 

I 2 - < 1L75 ) 

ij 


If only the largest singular value is taken into account, A is approximated by the 
rank— 1 matrix 


(i) 


As an example, consider a m x n matrix 
/ xi(ti) ... x n (t x ) \ 


A = 


\ x i ( t m ) - - - x n (t m ) J 


(11.76) 


(11.77) 


which contains the values of certain quantities X\ ... x„ observed at different times 
t\ ... t m . For convenience, we assume that the average values have been subtracted, 
such that x i = 0. Approximation (11.76) reduces the dimensionality to 1, 
i.e. a linear relation between the data. The i-th row of A, 


(*t (tj) ... x„(ti)) (11.78) 

is approximated by 

stMi.t (t»i,i ... u„,t) (11.79) 

which describes a direct proportionality of different observables 

Xj(ti) = —x k (td. (11.80) 

Vj, 1 v k ,i 

According to (11.75) this linear relation minimizes the mean square distance 
between the data points (11.78) and their approximation (11.79). 

Example: Linear approximation [131] 

Consider the data matrix 

T /l 2 3 4 5 \ 

A ~ \1 2.5 3.9 3.5 4.0 ) ' (1L81) 

First subtract the row averages 


x = 3 y = 2.98 


(11.82) 
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to obtain 


a t _( ~2 -1 0 1 2 \ 

V —1.98 -0.48 0.92 0.52 1.02/ ' 
Diagonalization of 


iT . ( 10.00 7.00 \ 

A J = ( 7.00 6.308 ) 

gives the eigenvalues 

d x = 15.393 d 2 = 0.915 


and the eigenvectors 


/ 0.792 —0.610\ 
y 0.610 -0.792 ) ' 


Since there are no zero singular values we find 


U = ATS" 1 


/ —0.181 —0.380 \ 
-0.070 0.252 
0.036 0.797 

0.072 -0.217 
v 0.143 —0.451 ) 


This gives the decomposition 4 


A = (m u 2 ) ^ Sl s ^j = Jiuiv[ + s 2 u 2 vj 


/-2.212 

—1.704 \ 


/ 0.212 —0.276 \ 

-0.860 

-0.662 


-0.140 0.182 

0.445 

0.343 

+ 

-0.445 0.577 

0.879 

0.677 


0.121 -0.157 

v 1.748 

1.347 y 


v 0.252 -0.327 y 


(11.83) 


(11.84) 


(11.85) 


( 11 . 86 ) 


(11.87) 


( 11 . 88 ) 


If we neglect the second contribution corresponding to the small singular value .v 2 we 
have an approximation of the data matrix by a rank — 1 matrix. The column vectors 
of the data matrix, denoted as x and y, are approximated by 


4 u, v ( r is the outer or matrix product of two vectors. 
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x 


Fig. 11.2 (Linear approximation by singular value decomposition) The data set (1 1 .8 1 ) is shown as 
circles. The linear approximation which is obtained by retaining only the dominant singular value 
is shown by the squares and the full line. It minimizes the mean square distance to the data points. 
Stars and the dashed line show the approximation by linear regression, which minimizes the mean 
square distance in vertical direction 


x = s 1 u 11 u 1 y = siv 2 iui (11.89) 

which describes a proportionality between x and y (Fig. 1 1.2). 


11.2.4 Linear Least Square Fit with Singular Value 
Decomposition 

The singular value decomposition can be used for linear regression [131]. Consider 
a set of data, which have to be fitted to a linear function 

y = co + c\X\ - b c n x„ ( 11.90) 

with the residual 

r l = c 0 + c iX; ]- b c n x Un -y t . (11-91) 

Let us subtract the averages 

n-r = Cl (x,, 1 - Xi)- b c„ (x,.„ - x„) - (yt - y) (11.92) 


which we write in matrix notation as 



11.2 Singular Value Decomposition 


/n - r\ / x u - x, . 

• %l,n \ 

tc x \ 

\r m -r) 

• Xm,n Xn / 

\c n) 

or shorter 




r = Xc-y. 

Now let us insert the full decomposition of X 
r = UEV T c-y. 

Since U is orthogonal 

U T r = EV T c - U T y = Ea - b 


( yi-y 

y m - y 


where we introduce the abbreviations 
a = V T c b = U T y. 

The sum of squared residuals has the form 


\r\ 2 = \U T r\ 2 


a,. \ b r \| 
■ | V 0 0/V a n-r/ \K-r)\ 


= \S r a r - b,| 2 + b 2 t _ r < \S r a r - b,.| 2 . 


Hence a„_ r is arbitrary and one minimum of So is given by 
a r = S~% a„ , = 0 
which can be written more compactly as 
a = E+b 

with the Moore-Penrose pseudoinverse [132] of E 

K 1 \ 
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(11.93) 


(11.94) 


(11.95) 


(11.96) 


(11.97) 


(11.98) 


(11.99) 


( 11 . 100 ) 


( 11 . 101 ) 








250 


11 Data Fitting 


Finally we have 

c = VZ + U T y = X + y 
where 

X+ = VX + U T 

is the Moore-Penrose pseudoinverse of X. 

Example 

The following data matrix has rank 2 



( —3 -4 —5\ 


/ 1.0 \ 


-2 -3 -4 


i.i 

X = 

0 0 0 

y = 

0 


2 3 4 


-1.0 


4 5j 


1-1-1/ 


( 11 . 102 ) 


(11.103) 


(11.104) 


A solution to the linear fit problem is given by 


c = X + y = 







/ 

1.0 \ 


-0.917 

1.167 

0 

-1.167 

0.917 \ 


1.1 

/ 0.525 

-0.167 

0.167 

0 

-0.167 

0.167 j 


0 

= [ 0.000 

0.583 

-0.833 

0 

0.833 

-0.583 / 


-1.0 

\-0.525 







-i-V 



The fit function is 
y = 0.525(X! - x 3 ) 


(11.105) 


(11.106) 


and the residuals are 

/ 0.05 \ 
-0.05 
0 

-0.05 
v 0.05 } 


Ac — y = 


(11.107) 
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11.2.5 Singular and Underdetermined Linear Systems 
of Equations 


SVD is also very useful to solve linear systems with a singular or almost singular 
matrix. Consider a system 


fan an A (x A f b A 


\ a m i ... a nm J \ x n J \ b m J 


(11.108) 


with n > m, i.e. more unknowns than equations. SVD transforms this system into 


/ Mil ■ 

• ^1 m \ 

^i 

o...o\ 

/Ihi • 

• Vnl\ 

f x i\ 

(bl\ 

\ l^ml • 

• / 

V 

S m 0 ... 0 y 

\dk • 

• Vnn ) 

\X„ / 

\ bm ) 


(11.109) 

Substituting 


/yi\ 

V3V 


till . 

• ^711 ^ 

f x A 

/ c ‘\ 

( MU • 

• U\ m \ 

/M 



: 

and I 

= : 


1 

V\n • 

• Vnn ) 

\x„ / 

\ Cm i 

\ 1 • 


\k) 


( 11 . 110 ) 


it remains to solve 

(s\ 0...0\ fyi\ f ci 


\ 


0...0J \y„J V 


( 11 . 111 ) 


For y i ... y m the solution is 

y, = sfc'i (11.112) 

whereas y m +i...y n are arbitrary and parametrize the solution manifold. Back 
substitution gives 


X = Xp + z 


( 11 . 113 ) 
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with the particular solution 


Dll - - ■ V\ m \ / 1 

V n \ ■■■ V nm / 


-1 


Mil ■ ■ • Wl m \ ' / i>l \ 


S m J V V fn | ... u mm J 


(11.114) 


, bm J 


and 


z = 


/ I’l.m+l ■ 

• ^1 n \ 

/ym+ 1 \ 

\ Vfi,m+ 11 • 

• Vnn ) 

V y* 1 ) 


(11.115) 


which is in the nullspace of A 


Az = UEV 1 V 


( ° \ 


l 0 \ 

ym +1 

= U (S 0) 

y>n+ 1 

v y« ) 


v y« ) 


= 0. 


(11.116) 


If m — r singular values are zero (or if the smallest singular values are set to zero) 
(11.111) becomes 

(s i 0...0\ 


!yi\ 


/°i\ 

\yn> 


^ ) 


s r 0... 0 

0 ... 0 0...0 


\0 ... 0 0...0/ 
which gives on the one hand 
y t = sj x a i = l... r 

yi = arbitrary i = r + 1... m 
but also requires 

Cj = 0 i = r + 1... m. 


(11.117) 


(11.118) 

(11.119) 

( 11 . 120 ) 


If this condition is not fulfilled, the equations are contradictory and no solution exists 
(e.g. if two rows of A are the same but the corresponding elements of b are different). 
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Problems 

Problem 11.1 Least Square Fit 

At temperatures far below Debye and Fermi temperatures the specific heat of a metal 
contains contributions from electrons and lattice vibrations and can be described by 

C(T) = aT + bT 3 . (11.121) 

The computer experiment generates data 

7) = T 0 +jAt (11.122) 

Cj = ( a 0 Tj + b 0 T?)(l + Ej) (11.123) 


with relative error 

ej = eij. (11.124) 

Random numbers are taken from a Gaussian normal distribution function 
(Sect. 9.2.6). 

The fit parameters a, b are determined from minimization of the sum of squares 
1 " 

S= -V (Cj-aTi-bT 3 ) 2 . (11.125) 

n ' 


Compare the “true values” ao, bo with the fitted values a, b. 


Chapter 12 

Discretization of Differential Equations 


Many processes in science and technology can be described by differen tial equations 
involving the rate of changes in time or space of a continuous variable, the unknown 
function. While the simplest differential equations can be solved exactly, a numer¬ 
ical treatment is necessary in most cases and the equations have to be discretized 
to turn them into a finite system of equations which can be solved by computers 
[133-135]. In this chapter we discuss different methods to discretize differential 
equations. The simplest approach is the method of finite differences, which replaces 
the differential quotients by difference quotients (Chap. 3 ). It is often used for the 
discretization of time. Finite difference methods for the space variables work best 
on a regular grid. Finite volume methods are very popular in computational fluid 
dynamics. They take averages over small control volumes and can be easily used 
with irregular grids. Finite differences and finite volumes belong to the general class 
of finite element methods which are prominent in the engineering sciences and use 
an expansion in piecewise polynomials with small support. Spectral methods, on the 
other hand, expand the solution as a linear combination of global basis functions 
like polynomials or trigonometric functions. A general concept for the discretization 
of differential equations is the method of weighted residuals which minimizes the 
weighted residual of a numerical solution. Most popular is Galerkin’s method which 
uses the expansion functions also as weight functions. Simpler are the point colloca¬ 
tion and subdomain collocation methods which fulfill the differential equation only 
at certain points or averaged over certain control volumes. More demanding is the 
least-squares method which has become popular in computational fluid dynamics 
and computational electrodynamics. The least-square integral provides a measure 
for the quality of the solution which can be used for adaptive grid size control. 

If the Green’s function is available for a problem, the method of boundary elements 
is an interesting alternative. It reduces the dimensionality and is, for instance, very 
popular in chemical physics to solve the Poisson-Boltzmann equation. 
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12 Discretization of Differential Equations 


12.1 Classification of Differential Equations 


An ordinary differential equation (ODE) is a differential equation for a function of 
one single variable, like Newton’s law for the motion of a body under the influence 
of a force field 


d 2 


m —x(f) = F (x,t), 
dt 2 


( 12 . 1 ) 


a typical initial value problem where the solution in the domain to < t < T is 
determined by position and velocity at the initial time 


x(t = to) = x 0 


— x(t = to) = v 0 . 
df 


( 12 . 2 ) 


Such equations of motion are discussed in Chap. 13. They also appear if the spatial 
derivatives of a partial differential equation have been discretized. Usually this kind 
of equation is solved by numerical integration over finite time steps At = f„+t — t n . 
Boundary value problems, on the other hand, require certain boundary conditions 1 to 
be fulfilled, for instance the linearized Poisson-Boltzmann equation in one dimension 
(Chap. 18). 


d 2 1 

- k 2 <P = --p(x) 
dx z £ 


(12.3) 


where the value of the potential is prescribed on the boundary of the domain .ro < 

x < Xl 


0(X O ) = <P O <P(Xi)=<P\. 


(12.4) 


Partial differential equations (PDE) finally involve partial derivatives with respect 
to at least two different variables, in many cases time and spatial coordinates. 

Linear Second Order PDE 

A very important class are second order linear partial differential equations of the 
general form 


N N 


! = 1 ;'=1 


& 2 

dxi dx j 




' dxi 


+ c 


f(x i ... Xn) + d — 0 


(12.5) 


'Dirichlet b.c concern the function values, Neumann b.c. the derivative, Robin b.c. a linear combi¬ 
nation of both, Cauchy b.c the function value and the normal derivative and mixed b.c. have different 
character on different parts of the boundary. 
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where the coefficients a,j , bj , c, d are functions of the variables x\ ... Xn but do not 
depend on the function / itself. The equation is classified according to the eigenvalues 
of the coefficient matrix a ( / as [136] 

Elliptical 

If all eigenvalues are positive or all eigenvalues are negative, like for the Poisson 
equation (Chap. 18) 


/ d 2 d 2 
\(9x 2 dy 2 dz 2 ) 


<P(x, y, z) 


1 

--g(x,y,z), 

£ 


( 12 . 6 ) 


Hyperbolic 

If one eigenvalue is negative and all the other eigenvalues are positive or vice versa, 
for example the wave equation in one spatial dimension (Chap. 20). 


d 2 


d 2 


W f - c dS f = 0 ' 

Parabolic 

If at least one eigenvalue is zero, like for the diffusion equation (Chap. 21) 


(12.7) 


d 

~dt 


f (x, y, z,t) — D 


/ d 2 d 2 d 2 \ 

\c)x 2 dy 2 dz 2 ) 


fix, y, z, t) = S(x, y, z, t). 


( 12 . 8 ) 


Ultra-Hyperbolic 

If there is no zero eigenvalue and more than one positive as well as more than one 
negative eigenvalue. Obviously the dimension then must be 4 at least. 

Conservation Laws 

One of the simplest first order partial differential equations is the ID advection 
equation 

d d 

— f(x,t) + u — f(x,t) = 0 (12.9) 

which describes transport of a conserved quantity with density / (for instance mass, 
number of particles, charge etc.) in a medium streaming with velocity u. This is a 
special case of the class of conservation laws (also called continuity equations) 

^-/(x, t ) + divj(x, t ) = g(x, t) (12.10) 

at 

which are very common in physics. Here J describes the corresponding flux and g 
is an additional source (or sink) term. For instance the advection-diffusion equation 
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(also known as convection equation) has this form which describes quite general 
transport processes: 

d 

— C = div (D grad C — uC) + S(x, t) = — div J + S(x, t) (12.11) 

at 

where one contribution to the flux 


J = —D grad C + u C 


( 12 . 12 ) 


is proportional to the gradient of the concentration C (Fick’s first law) and the second 
part depends on the velocity field u of a streaming medium. The source term S 
represents the effect of chemical reactions. Equation (12.11) is also similar to the 
drift-diffusion equation in semiconductor physics and closely related to the Navier 
Stokes equations which are based on the Cauchy momentum equation [137] 

9u \ 

— + u grad u I = diver + f (12.13) 

at ) 

where a denotes the stress tensor. Equation ( 12.1 0) is the strong or differential form of 
the conservation law. The requirements on the smoothness of the solution are reduced 
by using the integral form which is obtained with the help of Gauss’ theorem 



d 

Wt fM 


3(x, t)j 


dV 


J(x, t)dA = 0. 


dv 


(12.14) 


An alternative integral form results from Galerkin’s [138] method of weighted 
residuals which introduces a weight function w(x) and considers the equation 


d 


•'>) 


■ divj(x, t) — g(x, t) ) w(x) dV — 0 


(12.15) 


or after applying Gauss’ theorem 



J~/(x, t ) - ff(x, f)^ 

w(x)J(x, t)dA = 0. 


w(x) — J(x, t) gradw(x) 


dV 


(12.16) 


Jdv 

The so called weak form of the conservation law states that this equation holds for 
arbitrary weight functions w. 
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12.2 Finite Differences 


The simplest method to discretize a differential equation is to introduce a grid of 
equidistant points and to discretize the differential operators by finite differences 
(FDM) as described in Chap. 3. For instance, in one dimension the first and second 
derivatives can be discretized by 

x —> x m — mAx m = 1 ... M (12.17) 


fix) f m = f{x m ) nt= 1 ...M 


(12.18) 


d£ / d_ _ fm + l - fm ( d_ \ _ /,„+! - f m -1 

dx \<9x ) m Ax ° r \i9x ) m 2Ax 

d 2 f (&_ \ = fn + 1 + fm- 1 - 2 /„, 

dx 2 \9x 2 ) m Ax 2 


(12.19) 

( 12 . 20 ) 


These expressions are not well defined at the boundaries of the grid m = 1, M unless 
the boundary conditions are taken into account. For instance, in case of a Dirichlet 
problem /o and /m+i are given boundary values and 


(« f \ _ h ~ fo (&_ f \ _ h~ 2 /i + /o 

/j 2 Ax \chr 2 J 1 Ax 2 


( 12 . 21 ) 


( d_ _ /m+i - Im or Im +i — /m-i / (f_ ,\ 
ydx 1 ) M Ax ° r 2Ax \dx 21 ) M 


/m-i — 2/m + /m+i 
Ax 2 


( 12 . 22 ) 


Other kinds of boundary conditions can be treated in a similar way. 


12.2.1 Finite Differences in Time 

Time derivatives can be treated similarly using an independent time grid 

t —*■ t n = nAt n = 1... N (12.23) 

f(t,x)-> C = fit n ,x m ) (12.24) 


and finite differences like the first order forward difference quotient 
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or the symmetric difference quotient 

Q f fn+l _ ra-1 

u J J m J m 


dt 


2 At 


(12.26) 


to obtain a system of equations for the function values at the grid-points f ' n For 
instance for the diffusion equation in one spatial dimension 


df(x,t ) d 2 

— Wt — = D dx^^ X ’ + S(X ' 


(12.27) 


the simplest discretization is the FTCS (forward in time, centered in space) scheme 


(/,;; +1 - o = D ^c +l + si-x - 2 o +s n m A t 


At 


which can be written in matrix notation as 


f n+ , -f„ = D^Mf n + S n At 


(12.28) 


(12.29) 


with 



/fl\ 


/ —2 1 \ 


ft 


1 -2 1 

f„ = 

sn 

J 3 

and M = 

1 -2 1 


v/mJ 


V 1 -2/ 


(12.30) 


12.2.2 Stability Analysis 

Fully discretized linear differential equations provide an iterative algorithm of the 
type 2 

f„+i = Af„ + S„z\r (12.31) 

which propagates numerical errors according to 

fn+i + f-n+i = A( f„ + e„) + S„At (12.32) 


2 Differential equations which are higher order in time can be always brought to first order by 
introducing the time derivatives as additional variables. 
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e j +1 — Ae j- 


(12.33) 


Errors are amplified exponentially if the absolute value of at least one eigenvalue of 
A is larger than one. The algorithm is stable if all eigenvalues of A are smaller than 
one in absolute value (1.4). If the eigenvalue problem is difficult to solve, the von 
Neumann analysis is helpful which decomposes the errors into a Fourier series and 
considers the Fourier components individually by setting 


f n=9 n {k) 


V 


i kM 


(12.34) 


and calculating the amplification factor 


/, 


n +1 


fn 
J m 


= Ia(*)I ■ 


(12.35) 


The algorithm is stable if \g(k)\ < 1 for all k. 

Example For the discretized diffusion equation (12.28) we find 


g n+l (k) = g n (k) + 2D—g n (k) (cos k- 1) 

At At , / k\ 

g(k) = 1 + 2D^(cos* - 1) = 1 - 4Z^ sin j 

!- 4 ^ <,(.)<! 


(12.36) 

(12.37) 

(12.38) 


hence stability requires 


At 1 

D - T < -. 

Ax 2 2 


(12.39) 


12.2.3 Method of Lines 

Alternatively time can be considered as a continuous variable. The discrete values 
of the function then are functions of time (so called lines) 


fm(t) (12.40) 

and a set of ordinary differential equations has to be solved. For instance for diffusion 
in one dimension (12.27) the equations 
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d f m D 

—— = JX (fm +1 + fm -1 — 2 /,„) + 5 ,„(/) 

dr «- 

which can be written in matrix notation as 



// 1\ 
/1 


(-2 1 

1 -2 1 

\ 

d 

h 

D 

1 -2 1 


dr 

\fM ) 

Ax 2 

V 1 

-2/ 


or briefly 


(12.41) 


//i\ 


( sm 71 fo \ 

h 


s 2 

h 

+ 

S3 

\fM ) 


{Sm + tt/m+1 J 


(12.42) 


•j-f(f) = Af(f) + S(f). (12.43) 

dr 

Several methods to integrate such a semi-discretized equation will be discussed in 
Chap. 13. If eigenvectors and eigenvalues of A are easy available, an eigenvector 
expansion can be used. 


12.2.4 Eigenvector Expansion 

A homogeneous system 

j-f(0 = Am (12.44) 

dr 

where the matrix A is obtained from discretizing the spatial derivatives, can be solved 
by an eigenvector expansion. From the eigenvalue problem 


Af = Af (12.45) 

we obtain the eigenvalues A and eigenvectors f A which provide the particular solu¬ 
tions: 

m = e xt h (12.46) 


^{e x < t x ) = \{e xt f A ) = A{e x ' f A ). 
dr 


(12.47) 
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These can be used to expand the general solution 

f(r) = ^C A e A? f A . (12.48) 

A 

The coefficients C\ follow from the initial values by solving the linear equations 


f {t = 0) = £ C A f A. (12.49) 

A 

If the differential equation is second order in time 
d 2 

-^f(f) = Af(f) (12.50) 

df 2 

the particular solutions are 

f(t) = e ±,Vx f x (12.51) 

d 2 

^(e ±,V ~ x f A ) = \{e ±, ^ x f A ) = A(e ±l ^ f A ) (12.52) 

and the eigenvector expansion is 

f (0 = X ( C A+e ,VX + Cx-e- ,Vx ) f A . (12.53) 

A 


The coefficients C\± follow from the initial amplitudes and velocities 
f (f = 0) = X (C A+ + C A-) f A 

A 

Af( ? = 0) = Y, ^A( Cx+ - C A _)f A . (12.54) 

A 

For a hrst order inhomogeneous system 

y-f (0 = Am + S(0 (12.55) 

dr 

the expansion coefficients have to be time dependent 

f(r) = ^C A (r> A 'f A 


(12.56) 
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and satisfy 

d i d(7\ \. 

-m - Am = Y = sw. 

at at 

x 

After taking the scalar product with f M 3 

can be solved by a simple time integration. For a second order system 
d 2 

-jun = Am + s(o 
dr 2 

we introduce the first time derivative as a new variable 



to obtain a first order system of double dimension 

io-cao-ra 

where eigenvectors and eigenvalues can be found from those of A (12.45) 

(° o) (±y)=(y o = ±srx {±% x ) 

f l) (° J) = (AfI ±VAfI) =±VX(±VAf A r f[). 

Insertion of 

? c ^'(A) +c ^(-X) 

gives 

X^ dC >+ c VAf/ f A ^ dCx ~r^t( f A \ (S (t)\ 

VVAfJ + ^r e i-VAfJ-ioj 


3 If A is not Hermitian we have to distinguish left- and right-eigenvectors. 


(12.57) 

(12.58) 

(12.59) 

(12.60) 

(12.61) 

(12.62) 

(12.63) 


(12.64) 
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and taking the scalar product with one of the left-eigenvectors we end up with 
^ ± = ^(fAS(f))e“' /X ' (12.65) 

d ^ = -^(f A S(f))e VXj . (12.66) 

at 2 

12.3 Finite Volumes 

Whereas the finite differences method uses function values 

fij.k = f(xi, yj , Zk) (12.67) 

at the grid points 

r ijk = (Xi,yj,zk), (12.68) 

the finite volume method (FVM) [139] averages function values and derivatives over 
small control volumes V r which are disjoint and span the domain V (Fig. 12.1) 

V = |J V r V r p| V r > = 0Vr £ r'. (12.69) 



Fig. 12.1 (Finite volume method) The domain V is divided into small control volumes V r , in the 
simplest case cubes around the grid points r jjk 
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The averages are 

J=±[ dVf( r) (12.70) 

Jv r 


or in the simple case of cubic control volumes of equal size h? 

2 rXi+h/2 t-yj+h/2 r-Zk+h/2 


fijk = 73 / dx dy dzf(x,y,z). (12.71) 

n Jxi-h/2 J yj—h/2 Jzk-h/2 

Such average values have to be related to discrete function values by numerical 
integration (Chap. 4). The midpoint rule (4.17), for instance replaces the average by 
the central value 


f ijk = f(xi,yj,z k ) + 0(h 2 ) (12.72) 

whereas the trapezoidal rule (4.13) implies the average over the eight corners of the 
cube 


fijk = 


; Z 

m,n,p=± 1 


f 2 ) y.i+n/ 2 5 Zk+p/ 2 ) T 0(h ~). 


(12.73) 


In (12.73) the function values refer to a dual grid (Fig. 12.2) [139] centered around 
the vertices of the original grid (12.68). 


r I+ '/2,J+ '/-!,k+'/l 


( h h h 
[xi + -, yj + -,Z k +- 


) 


(12.74) 


Fig. 12.2 (Dual grid) The 
dual grid (black) is centered 
around the vertices of the 
original grid (red) 
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The average gradient can be rewritten using the generalized Stokes’ theorem as 

grad fijk — tj [ dV grad /(r) = cf f(r)dA. (12.75) 

V Jv ijk JdVijt 


For a cubic grid we have to integrate over the six faces of the control volume 


_ 1 

grad f ijk = 


/ fZk+h/2 

Jzt-h/2 

rZt+h/2 
Jzk-h/2 
rXj+h/2 
\ J.x'i —h/2 


dzfy y !- h/2 

dz - lx, 

dx L 


yj-h/2 
■Xi+h/2 
-h/2 
yj+h/2 
yj-h/2 


dy ( f(xi + 2 ,y,z)~ f{xi -\,y,z))\ 
dx ( f(xi, y+ \,z)~ f(xi, y - §, z)) 
dy ( f(xi, y, z + §) - f(x it y,z— |)) ) 


(12.76) 


The integrals have to be evaluated numerically. Applying as the simplest approxi¬ 
mation the midpoint rule (4.17) 


rXj+h/2 


dx 


lxi-h/2 


r yj+h/2 
' yj-h/2 


dy fix, y) = h 2 (/(x ; , yj) + 0(h 2 )) 


this becomes 


_ 1 

grad fj k = - 


/ f(xi + | ,y jt z k ) - f(xi -\,yj, Zk)\ 
fixj, yj + |, z k ) - fixi, yj - \, z k ) 

\f(Xi, yj, Zk + |) - f(Xj, yj, Zk-\)) 


(12.77) 


(12.78) 


which involves symmetric difference quotients. However, the function values in 
(12.78) refer neither to the original nor to the dual grid. Therefore we interpolate 
(Fig. 12.3). 


hi, . 

fixi ± -,yj,Zk) ^ 2 {f(xi+uyj, Zi) + f(x t -i, yj , z,)) 

1 ( h h \ 

- I fixt + -, yj, Zk) - fixi - yj , Zk I 

~ ^ (/ (x '+ 1 - y /- - fixi-i, yj, Zk) 


or 


/ (x, ± yj, z*^ » X / (x, ± 


m,n=± 1 


2’yj+ m 2’ Zk+n ^ 


h\ 

n— I . 

2 / 


(12.79) 


(12.80) 


(12.81) 


The finite volume method is capable of treating discontinuities and is very flexible 
concerning the size and shape of the control volumes. 
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Fig. 12.3 (Interpolation 
between grid points) 
Interpolation is necessary to 
relate the averaged gradient 
(12.78) to the original or 
dual grid 


x i-i/2 yj+ 1/2 

X i+1/2 Vj+1/2 


x i 

/l 


X i-1/2 

, y j-1/2 

X i+1/2 ( 

, y j-1/2 






12.3.1 Discretization of fluxes 


Integration of (12.10) over a control volume and application of Gauss’ theorem gives 
the integral form of the conservation law 

vf 3dA+ Wtl / fdv = 7 /° dv (12 - 82) 

which involves the flux J of some property like particle concentration, mass, energy 
or momentum density or the flux of an electromagnetic field. The total flux through 
a control volume is given by the surface integral 


0= <£ JdA (12.83) 

JdV 

which in the special case of a cubic volume element of size h? becomes the sum over 
the six faces of the cube (Fig. 12.4). 



j-Xi+h/2 

ryj+h/ 2 / 

( h \ 

( h \ 

■ dx j 

' dy [J 

z 1 x,y,z k + x 1 

- Jz 1 X, y, Zk - - ) 

Jxi-h/2 J 

yj-h/2 \ 

V 2/ 

\ 2 ) 

i-Xi+h/2 

rZk+h/2 / 

/ h > 

\ ( h \ 

+ / dx 

/ dz ( 

Jy\ x ’yj + 

- J- ( x,yj - -,z) 

Jxi-h/2 

Jz k -h/2 \ 

V 2 ) 

' \ 2 / 


[ Zk+h/2 ryj+ h ' 2 ( ( h \ / h 

+ dz dy I J x I Xi + -,y,z \ - J z \Xi - -,y,z 
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Fig. 12.4 Flux through a 
control volume 



The surface integral can be evaluated numerically (Chap. 4). Using the midpoint 
approximation (12.77) we obtain 

1 1 / 

— #(*/, yt,Zi ) = - (J z (xi, yj,Zk+ 1/2 - J z (xi, yj,Zk- y 2 ) 

+ J y (Xiy j+ 1 / 2 , z*) - 7 v (xi, yy_i/ 2 , z*) + J x (x i+ i /2 , yj,Zk ) - J x iXi~y 2 , yj , z*)) - 

(12.85) 


The trapezoidal rule (4.13) introduces an average over the four corners (Fig. 12.3) 


l‘Xi+h/2 ryj+h/ 2 

/ t/v / 

lxi-h/2 Jyj—h/2 


dy f{x, y) 


^0 z 

\ m,n==hl 


f (Xi+m/h yj+n/ 2 ) "1“ 0(h ) 


) 


( 12 . 86 ) 


which replaces the flux values in (12.85) by the averages 


Jx(Xi± 1/2, y/, zjt) 


7y (xi , y_,±i/2, z^) 


J z (Xi , V /- Z^i/2) 


1 

4 

1 

4 

1 

4 


^ J Z iXi±l/ 2 , yj+m/2, Zk+n/2) 
m,n=± 1 

Jz^i+m/h yZk+n/l) 

m,n=±l 

Jz^i+m/2i yj+n/2i Zk± l /i )• 

m,«=±l 


(12.87) 

( 12 . 88 ) 

(12.89) 


One advantage of the finite volume method is that the flux is strictly conserved. 
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12.4 Weighted Residual Based Methods 

A general method to discretize partial differential equations is to approximate the 
solution within a finite dimensional space of trial functions. 4 The partial differential 
equation is turned into a finite system of equations or a finite system of ordinary 
differential equations if time is treated as a continuous variable. This is the basis of 
spectral methods which make use of polynomials or Fourier series but also of the 
very successful finite element methods. Even finite difference methods and finite 
volume methods can be formulated as weighted residual based methods. 

Consider a differential equation 5 on the domain V which is written symbolically 
with the differential operator T 

T[u(r)] = f(r) reV (12.90) 

and corresponding boundary conditions which are expressed with a boundary oper¬ 
ator B 6 

B[u{ r)] = g(r) r e dV. (12.91) 

The basic principle to obtain an approximate solution u (r) is to choose a linear 
combination of expansion functions N, (r) i = I ... r as a trial function which 
fulfills the boundary conditions 7 

r 

u = ^ j u i N i {r) (12.92) 

1=1 


B[u(r)] = g(r). (12.93) 

In general (12.92) is not an exact solution and the residual 

^(r) = T [m] (r) — /(r) (12.94) 

will not be zero throughout the whole domain V. The function u has to be deter¬ 
mined such that the residual becomes “small” in a certain sense. To that end weight 
functions 8 Wj j = 1... r are chosen to define the weighted residuals 


4 A1so called expansion functions. 

5 Generalization to systems of equations is straightforward. 

6 One or more linear differential operators, usually a combination of the function and its first deriv¬ 
atives. 

7 This requirement can be replaced by additional equations for the w/, for instance with the tau 
method [140], 

8 A1so called test functions. 



12.4 Weighted Residual Based Methods 


271 


Rj im .. ,u r ) = J dv Wj(r) (T[u\ (r) - /(r)) . (12.95) 

The optimal parameters w, are then obtained from the solution of the equations 
Rj(u\ ... u r ) = 0 j = 1... r. (12.96) 

In the special case of a linear differential operator these equations are linear 

dVwj(r)T[Ni(r)]- 

Several strategies are available to choose suitable weight functions. 


j dV Wj(r)f(r) = 0. (12.97) 



12.4.1 Point Collocation Method 

The collocation method uses the weight functions wj( r) = <5 (r — r ; j, with certain 
collocation points r j e V. The approximation u obeys the differential equation at 
the collocation points 

o = r j = nmj) - firj) ( 12 . 98 ) 

and for a linear differential operator 

r 

0 = T[Ni\{rj) - f (vj ). (12.99) 

i=i 

The point collocation method is simple to use, especially for nonlinear problems. 
Instead of using trial functions satisfying the boundary conditions, extra collocation 
points on the boundary can be added (mixed collocation method). 


12.4.2 Sub-domain Method 

This approach uses weight functions which are the characteristic functions of a set 
of control volumes V, which are disjoint and span the whole domain similar as for 
the finite volume method 

V = U V 1 V J n v r = ()V ' * j' 

j 


( 12 . 100 ) 
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wj( r) 


1 r eVj 
0 else. 


The residuals then are integrals over the control volumes and 
0 =Rj= f dV (T[«](r)-/(r)) 

JVj 


respectively 

0 = [ dVT[N t U r)- [ dVf( r). 

/ J Vj JVj 


( 12 . 101 ) 


( 12 . 102 ) 


(12.103) 


12.4.3 Least Squares Method 


Least squares methods have become popular for first order systems of differential 
equations in computational fluid dynamics and computational electrodynamics [141, 
142], 

The L2-norm of the residual (12.94) is given by the integral 


S = I dVRir) 2 . (12.104) 

Jv 

It is minimized by solving the equations 
dS f dR 

0=j— = 2 dV — R(r) (12.105) 

Oil j J V OU j 

which is equivalent to choosing the weight functions 


Wj (r) 


dR 

duj 


R(r) = 



duj 


ih N, (r) 


or for a linear differential operator simply 


Wj(r) = T [Nj (r)]. 


(12.106) 


(12.107) 


Advantages of the least squares method are that boundary conditions can be 
incorporated into the residual and that S provides a measure for the quality of the 
solution which can be used for adaptive grid size control. On the other hand S involves 
a differential operator of higher order and therefore much smoother trial functions 
are necessary. 
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12.4.4 Galerkin Method 


Galerkin’s widely used method [138, 143] chooses the basis functions as weight 
functions 


Wj( r) = Nj (r) 

and solves the following system of equations 


(12.108) 


dV Nj(r)T 


y' J u i N i ( r) 

_ i 

or in the simpler linear case 

2> [ dV Nj(r)T[Ni(r)] = f dV JV,-(r)/(r). 


J dV Nj(r)f(r) = 0 (12.109) 


( 12 . 110 ) 


12.5 Spectral and Pseudo-Spectral Methods 

Spectral methods use basis functions which are nonzero over the whole domain, the 
trial functions being mostly polynomials or Fourier sums [144]. They can be used to 
solve ordinary as well as partial differential equations. The combination of a spectral 
method with the point collocation method is also known as pseudo-spectral method. 


12.5.1 Fourier Pseudo-Spectral Methods 

Linear differential operators become diagonal in Fourier space. Combination of 
Fourier series expansion and point collocation leads to equations involving a dis¬ 
crete Fourier transformation, which can be performed very efficiently with the Fast 
Fourier Transform methods. 

For simplicity we consider only the one-dimensional case. We choose equidistant 
collocation points 

x m — mAx m = 0, 1... M — 1 (12.111) 

and expansion functions 

Nj(x) = e ikjX kj = -^-j j =0, 1...M— 1. (12.112) 

MAx 


For a linear differential operator 
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C[e k ' x } = l(kj)e' klX (12.113) 

and the condition on the residual becomes 

M -1 

0 = R m = X u i Kkj)e ikiXm - f(x m ) (12.114) 

7=0 

or 

M -1 

f(x m ) = X U J l(kj)e i2nmj/M (12.115) 

7=0 

which is nothing but a discrete Fourier back transformation (Sect. 7.2, 7.19) which 
can be inverted to give 

1 M -1 

Uj l(kj ) = - X f(xm) e- i2 ^ /M . (12.116) 

m= 0 

Instead of exponential expansion functions, sine and cosine functions can be used 
to satisfy certain boundary conditions, for instance to solve the Poisson equation 
within a cube (Sect. 18.1.2). 


12.5.2 Example: Polynomial Approximation 

Let us consider the initial value problem (Fig. 12.5) 
d 

-u{x)-u{x) = 0 w(0) = 1 for 0 < x < 1 (12.117) 

d.r 

with the well known solution 

u(x) — e v . (12.118) 

We choose a polynomial trial function with the proper initial value 

u(x) = 1 + uix + U2X 2 . (12.119) 

The residual is 

R(x) = u\+2u2X — (l + u\X + U 2 X 2 ) = («i — 1) + (2 m2 — u\)x — u^x 2 . (12.120) 
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Fig. 12.5 (Approximate 
solution of a simple 
differential equation) The 
initial value problem 
^u(x) — u(x ) = 0 
ti(0) =1 for 0 < x < 1 is 
approximately solved with a 
polynomial trial function 
u(x) = 1 + u\x + U 2 X 2 . The 
parameters 1 * 1,2 are 
optimized with the method 
of weighted residuals using 
point collocation (full curve), 
sub-domain collocation 
(dotted curve), Galerkin’s 
method ( dashed curve) and 
least squares (dash-dotted 
curve). The absolute error 
u(x) — e* (Top) and the 
residual 

R(x) = 37«(x) - u(x) = 

(111 — l) + (2un — Ul)x — U2X 2 
both are smallest for the least 
squares and sub-domain 
collocation methods 



x 


12.5.2.1 Point Collocation Method 


For our example we need two collocation points to obtain two equations for the two 
unknowns 1 * 1 , 2 - We choose x\ = 0, xo = Then we have to solve the equations 


R(xi) = u\ — 1 = 0 
1 3 

= -Ml + -M2 -1=0 
which gives 

2 

Mi = 1 M2 = — 

■ 3 


( 12 . 121 ) 

( 12 . 122 ) 


(12.123) 


u c = 1 + x 


(12.124) 
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12.5.2.2 Sub-domain Method 

We need two sub-domains to obtain two equations for the two unknowns u\ 2 - We 
choose Vi = {x,0<jc < \}, V 2 — {x, ^ < x < 1}. Integration gives 


Ri = ~u 1 T — m 2 - 

- = 0 

(12.125) 

8 24 

2 


1 11 

1 


Rl = —Ml H-M2 — 

“ 8 24 

2=° 

(12.126) 

6 

«1 = M 2 = - 


(12.127) 

1 ^ 6 

=1+T'+ 7jX 

2 

(12.128) 


12.5.2.3 Galerkin Method 

Galerkin’s method uses the weight functions w\(x) = x, W 2 (x ) = x 2 . The equations 


, ‘ 1 15 1 

dx w\{x)R(x) = -u 1 H- 112 -=0 

) 6 12 2 

^ 13 1 

dx W 2 (x)R(x) = —u 1 H- «2 -=0 


12 


10 


have the solution 


u 1 = — Ul = — 
11 11 


10 

r 

10 


l!n — lT — x T — X . 

11 11 


(12.129) 

(12.130) 


(12.131) 

(12.132) 


12.5.2.4 Least Squares Method 

The integral of the squared residual 

f 1 2 4 1 2 1 8 2 

S= dx R(xY = 1 — mi -M2 H—Mj H— 11 1 M2 H-Mj 

Jo 3 3 2 15 

is minimized by solving 


dS 
du 1 


2 1 

-mi H—M2 —1=0 

3 2 


(12.133) 


(12.134) 
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as 1 16 4 

-— = -It 1 + —U 2 



(12.135) 


which gives 



(12.136) 


(12.137) 


12.6 Finite Elements 

The method of finite elements (FEM) is a very flexible method to discretize partial 
differential equations [145, 146]. It is rather dominant in a variety of engineering 
sciences. Usually the expansion functions N, are chosen to have compact support. 
The integration volume is divided into disjoint sub-volumes 


r 



(12.138) 


The Nj (x) are piecewise continuous polynomials which are nonzero only inside Vj 
and a few neighbor cells. 


12.6.1 One-Dimensional Elements 

In one dimension the domain is an interval V = [x; a < x < b] and the sub-volumes 
are small sub-intervals Vi = [x; x,- < x < x,+i}. The one-dimensional mesh is the 
set of nodes {a = xq, x\ ... x r = b}. Piecewise linear basis functions (Fig. 12.6) are 
in the 1-dimensional case given by 


Ni(x) = 


- v f+i —- v 

*1+1 -*> 

X—Xj-i 

Xj—Xj-l 


for xi < x < x/ + 1 

for X, _ j < x < x, 
0 else 


(12.139) 


and the derivatives are (except at the nodes x, ) 


forx; < x < X; + i 

forx,_i < x < Xj 
0 else 


N[(x) = 


Xj—Xi-l 


(12.140) 
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N o N m Nj 



Fig. 12.6 (Finite elements in one dimension) The basis functions Ni are piecewise continuous 
polynomials and have compact support. In the simplest case they are composed of two linear 
functions over the sub-intervals x;_i < x < x, and x; < x < x;+i 


Fig. 12.7 (Triangulation of y 

a two dimensional domain) 

A two-dimensional mesh is 
defined by a set of node 
points which can be regarded 
to form the vertices of a 
tri angulation 



12.6.2 Two-and Three-Dimensional Elements 

In two dimensions the mesh is defined by a finite number of points ( x y,j e V (the 
nodes of the mesh). There is considerable freedom in the choice of these points and 
they need not be equally spaced. 


12.6.2.1 Triangulation 

The nodes can be regarded as forming the vertices of a triangulation 9 of the domain 
V (Fig. 12.7). 

The piecewise linear basis function in one dimension (12.139) can be generalized 
to the two-dimensional case by constructing functions /V, (x, y) which are zero at all 
nodes except (x,-, y, ) 

N i {x j ,y j ) = 5 uj . (12.141) 


9 The triangulation is not determined uniquely by the nodes. 
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Fig. 12.8 (Finite elements in two dimensions) The simplest finite elements in two dimensions are 
piecewise linear functions N, (x , y) which are non-vanishing only at one node (x;, y;) (Right side). 
They can be constructed from small pyramids built upon one of the triangles that contains this node 
(Left side) 


These functions are linear over each triangle which contains the vertex i and can 
be combined as the sum of small pyramids (Fig. 12.8). Let one of the triangles be 
denoted by its three vertices as 7}^. 10 The corresponding linear function then is 

n ljk (x, y) = a + 0 x (x - x t ) + (3 y (y - y,) (12.142) 

where the coefficients follow from the conditions 


n i jk (xi, yi) = 1 n ijk (Xj,yj) = n ,■ jk (x k , y k ) = 0 


(12.143) 


as 


a = 1 f3 x = 


yj - yk 


2A, 


ijk 


Xk ~ Xj 

2 A, 


with 


(12.144) 


Aijk 



Xj — Xi X k — Xi 

yj - yt yk - yi 


(12.145) 


which, apart from sign, is the area of the triangle T,j k . The basis function N\ now is 
given by 


N t (x, y) 


riijk(x,y) (x , y) e T l]k 
0 else 


In three dimensions we consider tetrahedrons (Fig. 12.9) instead of triangles. The 
corresponding linear function of three arguments has the form 


n i,j, k ,i(x, y,z) = a + ( 3 x (x - Xi) + (3y(y - y t ) + P z (z - zd 


(12.146) 


10 The order of the indices does matter. 
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Fig. 12.9 (Tetrahedron) The 
tetrahedron is the 
three-dimensional case of an 
Euclidean simplex, i.e. the 
simplest polytop 


i 



and from the conditions ntjxiiXi, yi, Zi ) = 1 and riij ik ,i = 0 on all other nodes we 
find (an algebra program is helpful at that point) 

a — 1 


fix = 
Py = 

Pz = 


1 


6V UU 

1 

6 vjk, 

1 

'ijki 


det 


6 V, 


det 


det 


yk - yj yi - yj 

Zk Zj Zl Zj 

Zk Zj Zl Zj 
Xk ~ Xj X/ — Xj 

X k ~ Xj Xi - Xj 

yk - yj yi - yj 


(12.147) 


where V,^/ is, apart from sign, the volume of the tetrahedron 


Vtjki = - det 


Xj — Xi Xk — Xi xi — Xi 

yj - yi yk - yi yi - yt 

Zj Zi Zk Zi Zl Zi 


(12.148) 


12.6.2.2 Rectangular Elements 

For a rectangular grid rectangular elements offer a practical alternative to triangles. 
Since equations for four nodes have to be fulfilled, the basic element needs four 
parameters, which is the case for a bilinear expression. Let us denote one of the 
rectangles which contains the vertex i as Ri,j,k,i- The other three edges are 

(Xj , yj) = (x, + b x , yi) (x k , y k ) = (x ; , y, + b y ) (x h yf) = (x,- + b x , yi + b y ) 

(12.149) 

where b x = ±/j v , b y = ±/z v corresponding to the four rectangles with the common 
vertex i (Fig. 12.10). 
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x, y, x k yk 


t 



Fig. 12.10 (Rectangular elements around one vertex) The basis function IV; is a bilinear function 
on each of the four rectangles containing the vertex (x;, y;) 



Fig. 12.11 (Bilinear elements on a rectangular grid) The basis functions IV; (x, y) on a rectangular 
grid (Right side) are piecewise bilinear functions (Left side), which vanish at all nodes except 

O i, yi) 


The bilinear function (Fig. 12.1 1) corresponding to is 
riijxiix, y) = a + f3(x - X;) + 7 (y - >’,) + r](x - x,)(y - >’,). (12.150) 

It has to fulfill 


nij,k,i(Xi, y^ = 1 riijxi(Xj, yj) 


n ij,k,i(x k , y k ) = nijk,i(xi, yi) = 0 

(12.151) 


from which we find 


a = 1 (3 = — 


1 

b x 


1 



1 

b x by 


(12.152) 


X — X; 


y - yi 

by 


(X - X/) (y - yi) 


n i.j.k.l(x, y) = 1 - 


b 


b 


(12.153) 
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Fig. 12.12 (Three- 
dimensional rectangular 
grid) The basis function N, 
is trilinear on each of the 
eight cuboids containing the 
vertex i . It vanishes on all 
nodes except (x,-, _v;, z,) 
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The basis function centered at node i then is 
Nj(x, y) = 


nij,kj(x , y) (x, y ) e Rijxi 

0 else 


(12.154) 


Generalization to a three dimensional grid is straightforward (Fig. 12.12). We 
denote one of the eight cuboids containing the node (x,-, y t , 74 ) as with 

Xh’ z.ti) = ( x i + b x , y t ,Zi ) ■ ■ - (x h , yj 7 ,Zj 7 ) = (x,- + b x , y t + h v , z t + b z ). 
The corresponding trilinear function is 




= 1 


X - Xi y -y t z- Zi 


(x - Xi) (v - yi) + (x - Xi) (z - Zi) (z - Zj) (y - yt) 
b x by b x b z b z by 

(x - x^ (y - yi) (z - Zi) 


(12.155) 


12.6.3 One-Dimensional Galerkin FEM 


As an example we consider the one dimensional linear differential equation (12.5) 
d 2 . d 


\ 9x2 + b ~g^ + C J = 

in the domain 0 < x < 1 with boundary conditions 


(12.156) 


m( 0) = m(1) = 0. 


(12.157) 
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We use the basis functions from (12.139) on a one-dimensional grid with 


*i+t - Xi = hi 


(12.158) 


and apply the Galerkin method [147]. The boundary conditions require 


uq = UN —i = 0. 


(12.159) 


The weighted residual is 


0 = Rj = 



dx Nj (x) 


/ O 2 d \ 

(.”S5 +i, & + 7 



dx Nj(x)f(x). 


(12.160) 


First we integrate 


hj+hj-i 


r x *+1 

Nj (x)Nj (x) dx = / Nj (x)Nj(x)dx = 
J Xi -1 


Integration of the first derivative gives 

' 0 j = i 

/ dx Nj(x)N'Xx) = 

Jo —■ 

0 else 

For the second derivative partial integration gives 


J = i 


I j=i +1 

hi—i • • i 

J = 1 ~ 1 
0 \i-j\ > 1 


\ j = i- 1 
j = i +1 


/o 


•1 g2 

dxNj(x ) q x 2 N ‘M 


(12.161) 


(12.162) 


= Nj(l)N'(l-£)-Nj(0)N'(0 + e)- [ dx N'(x)N' [ (x) (12.163) 

Jo 

where the first two summands are zero due to the boundary conditions. Since 
Ni and Nj are nonzero only for x,_i < x < x, + \ we find 

hh J=i~ l 


- 1 d 2 

dx Nj(x) —Ni(x) = 


/o 


dx N'Ax ) Nj(x) = 


i = J 

h J = 1 + 1 

0 else 


(12.164) 
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Integration of the last term in (12.160) gives 


[ dx Nj(x)f(x) = [ dx Nj(x)f (x) 

70 Jxi -1 


dx 


*j ~ x i ~i 


~f(x)+ dx 


x i +1 


-fix). 


(12.165) 


Applying the trapezoidal rule 11 for both integrals we find 


rxj+i 


dxNj{x)fix) ~ f(xj) 


hi + hi -1 


(12.166) 


The discretized equation finally reads 


1 

h j -1 
ft 


M,_r 


(ft; + ft;—! ) 


+ C 


1 

2 
ft;l 


~~ u j -1 + 2 M f +1 


M; + — M;+l 

;-i/ “f 


ft i 


ft ; A ft ; _! /1; 

Uj~i H---M; + —M;+l 


= /(■*,) 


6 J 3 

ft; + ft;—! 


(12.167) 


which can be written in matrix notation as 
Au = Bf 


(12.168) 


where the matrix A is tridiagonal as a consequence of the compact support of the 
basis functions 


/ —1-L _L 

111 ho ’ 111 


A = a 


l 

V i ’ 


j_t_ j_ 

hj hj-i ’ hj 


hN-3 ’ h N _2 h N _o ) 


11 Higher accuracy can be achieved, for instance, by Gaussian integration. 
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/O’ k 


-i 0 1 

2 ’ u ’ 2 


■ 5 . 0 / 


+ c 


(lll+/lo) *1, 
3 ’ 6 


lij-i (/ij+/ij-i) 

6 ’ 3 ’ 6 


1^-3 f^JV-Z+^A'-3) 


/? o+/n 


B 




■ 

2 


h N -2+h N - 3 


(12.169) 


For equally spaced nodes h, — /z,_ 1 = h and after division by h (12.169) reduces to 
a system of equations where the derivatives are replaced by finite differences (12.20) 


1 1 

a—rM 2 + b-M ! + cMq u = f 
h~ h 

with the so called consistent mass matrix 


(12.170) 




M 0 = 


121 

6 3 6 


(12.171) 




V 
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and the derivative matrices 




\ /■■ 


\ 


M x = 


2 


0 


2 


m 2 = 


1 -2 1 


(12.172) 


V 




V 




The vector u is replaced by 


1 ' 

Mou = 1 H—M 2 u. 
6 


(12.173) 


Within the framework of the finite differences method 



(12.174) 


hence replacing it by uj (this is called mass lumping) introduces an error of the order 

0(h 2 ). 

12.7 Boundary Element Method 

The boundary element method (BEM) [148, 149] is a method for linear partial differ¬ 
ential equations which can be brought into boundary integral form 12 like Laplace’s 
equation (Chap. 18) 13 

-A<J(r) = 0 (12.175) 

for which the fundamental solution 
AG(r, r') = —<5(r — r') 
is given by 



in three dimensions 


(12.176) 


12 This is only possible if the fundamental solution or Green’s function is available. 
13 The minus sign is traditionally used. 
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G(r — r') = — In- in two dimensions. (12.177) 

27t |r — r'| 

We apply Gauss’s theorem to the expression [150] 
div [G(r — r')grad(0(r)) — <£(r)grad(G(r — r'))] 

=-<Z>(r)A(G(r-r')). (12.178) 


Integration over a volume V gives 



G(r-r') 


d_ 

dn 


(0(r))-0(r) 


d_ 

dn 


(G (r - r')) 


) 



dV (<2>(r)A(G(r-r')) = <Z>(r'). 


(12.179) 


This integral equation determines the potential self-consistently by its value and 
normal derivative on the surface of the cavity. It can be solved numerically by dividing 
the surface into a finite number of boundary elements. The resulting system of linear 
equations often has smaller dimension than corresponding finite element approaches. 
However, the coefficient matrix is in general full and not necessarily symmetric. 



Chapter 13 

Equations of Motion 


Simulation of a physical system means to calculate the time evolution of a model 
system in many cases. We consider a large class of models which can be described 
by a first order initial value problem 


— =f(Y(t),t ) Y(t = 0) = y 0 
dt 


(13.1) 


where Y is the state vector (possibly of very high dimension) which contains all 
information about the system. Our goal is to calculate the time evolution of the state 
vector Y (t ) numerically. For obvious reasons this can be done only for a finite number 
of values of t and we have to introduce a grid of discrete times t„ which for simplicity 
are assumed to be equally spaced 1 : 


(13.2) 


1«+1 — ^ 3 ^ 7 . 


Advancing time by one step involves the calculation of the integral 



(13.3) 


which can be a formidable task since f (Y (t), t) depends on time via the time depen¬ 
dence of all the elements of Y ( t ). In this chapter we discuss several strategies for the 
time integration. The explicit Euler forward difference has low error order but is use¬ 
ful as a predictor step for implicit methods. A symmetric difference quotient is much 
more accurate. It can be used as the corrector step in combination with an explicit 
Euler predictor step and is often used for the time integration of partial differen¬ 
tial equations. Methods with higher error order can be obtained from a Taylor series 
expansion, like the Nordsieck and Gear predictor-corrector methods which have been 
often applied in molecular dynamics calculations. Runge-Kutta methods are very 
important for ordinary differential equations. They are robust and allow an adaptive 

1 Control of the step width will be discussed later. 
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control of the step size. Very accurate results can be obtained for ordinary differential 
equations with extrapolation methods like the famous Gragg-Bulirsch-Stoer method. 
If the solution is smooth enough, multistep methods are applicable, which use infor¬ 
mation from several points. Most known are Adams-Bashforth—Moulton methods 
and Gear methods (also known as backward differentiation methods), which are 
especially useful for stiff problems. The class ofVerlet methods has been developed 
for molecular dynamics calculations. They are symplectic and time reversible and 
consen’e energy over long trajectories. 


13.1 The State Vector 

The state of a classical N-particle system is given by the position in phase space, or 
equivalently by specifying position and velocity for all the N particles 

Y= (rj,vi, ...,r N , \ N ). (13.4) 

The concept of a state vector is not restricted to a finite number of degrees of free¬ 
dom. For instance a diffusive system can be described by the particle concentrations 
as a function of the coordinate, i.e. the elements of the state vector are now indexed 
by the continuous variable x 

T=(ci(x),...cjif(x)). (13.5) 

Similarly, a quantum particle moving in an external potential can be described by 
the amplitude of the wave function 

Y =(*(%)). (13.6) 

Numerical treatment of continuous systems is not feasible since even the ultimate 
high end computer can only handle a finite number of data in finite time. Therefore 
discretization is necessary (Chap. 12), by introducing a spatial mesh (Sects. 12.2, 
12.3, 12.6), which in the simplest case means a grid of equally spaced points 

Xyx — ( ih.jh , kli) i = 1..i ma x■ j — l..y' max , k — l..k max (13.7) 

Y = ( cfXijk )... c M (Xyx-)) (13.8) 

Y=(*(x ijk )) (13.9) 

or by expanding the continuous function with respect to a finite set of basis functions 
(Sect. 12.5). The elements of the state vector then are the expansion coefficients 
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N 

\V >=Y J Cs\Vs> (13.10) 

s= 1 

Y = (Ci,..., C^). (13.11) 

If the density matrix formalism is used to take the average over a thermodynamic 
ensemble or to trace out the degrees of freedom of a heat bath, the state vector instead 
is composed of the elements of the density matrix 

N N N N 

>< = >< ( 13 - 12 > 
J=1 S ,= 1 5=1 S'= 1 


Y — (p ii • ■ • Pin, pn ■ ■ ■ Pin, ■ ■ ■ , Pn t • ■ ■ Pnn) ■ 


(13.13) 


13.2 Time Evolution of the State Vector 

We assume that all information about the system is included in the state vector. Then 
the simplest equation to describe the time evolution of the system gives the change 
of the state vector 

dT 

-t: =f(Y,t) (13.14) 

at 

as a function of the state vector (or more generally a functional in the case of a 
continuous system). Explicit time dependence has to be considered for instance to 
describe the influence of an external time dependent field. 

Some examples will show the universality of this equation of motion: 

• N-particle system 

The motion of N interacting particles is described by 

= (ri, Vi • ••) = (vi.ai •••) (13.15) 

at 

where the acceleration of a particle is given by the total force acting upon this particle 
and thus depends on all the coordinates and eventually time (velocity dependent 
forces could be also considered but are outside the scope of this book) 


F;( r ! • ■ • r N, t ) 


a, = 


m; 


(13.16) 
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• Diffusion 

Heat transport and other diffusive processes are described by the diffusion equation 

%=DAf + S(x,t) (13.17) 

ot 

which in its simplest spatially discretized version for 1-dimensional diffusion reads 
<9/(x,) D 

= — 2 (/'(x i+ i) +/(x,- 1 ) - 2/(x;)) + S(x„ t ). (13.18) 

• Waves 

Consider the simple 1 -dimensional wave equation 


dt 2 


,Pf 

dx 2 


(13.19) 


which by introducing the velocity g(x) = f(x) as an independent variable can be 
rewritten as 


d ( , d 2 \ 

- (f(x), g(x)) = l g(x), c 2 ^/(x) j . (13.20) 

Discretization of space gives 

(/'(Xi), g(Xi)) = ^g(Xj), (f(x i+1 ) +/(x,_ 1 ) - 2/(x ; ))j . (13.21) 


• two-state quantum system 

The Schroedinger equation for a two level system (for instance a spin -'/2 particle in 
a magnetic field) reads 


1 (CA = (H n (t) H n (t)\ /CA 
dr\C2/ \Hu(t) H 2 2(t))\Ci) 


(13.22) 


13.3 Explicit Forward Euler Method 


The simplest method which is often discussed in elementary physics textbooks 
approximates the integrand by its value at the lower bound (Fig. 13.1): 


Y(t n+1 )-Y(t„)^f(Y(t n ),t„)At. 


(13.23) 
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Fig. 13.1 Explicit Euler 
method 


f(t) 



n 


n+1 


The truncation error can be estimated from a Taylor series expansion 



(13.24) 


The explicit Euler method has several serious drawbacks 

• low error order 

Suppose you want to integrate from the initial time to to the final time to + T. For a 
time step of At you have to perform N = T/At steps. Assuming comparable error 
contributions from all steps the global error scales as NAt 2 = O(At). The error gets 
smaller as the time step is reduced but it may be necessary to use very small At to 
obtain meaningful results. 

• loss of orthogonality and normalization 

The simple Euler method can produce systematic errors which are very inconvenient 
if you want, for instance, to calculate the orbits of a planetary system. This can be 
most easily seen from a very simple example. Try to integrate the following equation 
of motion (see Example 1.5 on p. 13): 


dz 

— = i wz. 
d t 


(13.25) 


The exact solution is obviously given by a circular orbit in the complex plane: 


z = zoe 


(13.26) 


]z| = Izol = const. 


(13.27) 
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Fig. 13.2 Systematic errors 
of the Euler method 



Application of the Euler method gives 

z(t„+ 1 ) = z(t n ) + i uAtz(t„) — (1 + i u>At)z(t n ) (13.28) 


and you find immediately 

\z(t n )\ = Vl+wMr 2 = (1 +u 2 At 2 ) n/2 k(?o)l (13.29) 


which shows that the radius increases continually even for the smallest time step 
possible (Fig. 13.2). 

The same kind of error appears if you solve the Schroedinger equation for a 
particle in an external potential or if you calculate the rotational motion of a rigid 
body. For the N-body system it leads to a violation of the conservation of phase space 
volume. This can introduce an additional sensitivity of the calculated results to the 
initial conditions. Consider a harmonic oscillator with the equation of motion 


d 

df 


/x(t)\ / v(t) \ 

\v(t) J \-uj 2 x(t)) ' 


Application of the explicit Euler method gives 

(x(t + At)\ = (x(t)\ ( v(t) \ 
\v(t + At) J \v(t) J \-uj 2 x(t) J 


(13.30) 


(13.31) 


The change of the phase space volume (Fig. 13.3) is given by the Jacobi determinant 


d(x(t + At), v(t + At)) 


1 At 

8(x(t), v(t)) 


-J 1 At 1 


1 + (uj At) 2 . 


(13.32) 


In this case the phase space volume increases continuously. 
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Fig. 13.3 Time evolution of 
the phase space volume 


Fig. 13.4 Implicit backward 
Euler method 



13.4 Implicit Backward Euler Method 

Alternatively let us make a step backwards in time 

Y(t„) - Y(h+i ) « -f(Y(t n+ 1), t n+1 )At (13.33) 

which can be written as (Fig. 13.4) 

Y(t n+ 1) « Y(t n ) +f(Y(t n+1 ), t n+ i)At. (13.34) 

Taylor series expansion gives 

Y(tn) = Y(t n+ 1 ) - ^Y{t n+l )At + ^F(r„ +1 )^ + ■ • ■ (13.35) 

at at 1 2 


which shows that the error order again is 0(At 2 ). The implicit method is sometimes 
used to avoid the inherent instability of the explicit method. For the examples in 
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Sect. 13.3 it shows the opposite behavior. The radius of the circular orbit as well 
as the phase space volume decrease in time. The gradient at future time has to be 
estimated before an implicit step can be performed. 


13.5 Improved Euler Methods 

The quality of the approximation can be improved significantly by employing the 
midpoint rule (Fig. 13.5) 

Y(t n+ 1) - Y(tn) (t (r + y) , tn + y) At. (13.36) 

The error is smaller by one order of At: 

Y(t n ) +f (y + v) ’ tn + y) At 

/d Y Atd 2 Y \ 

= Y(t n ) + y — (t n ) + — + ---J At 

= Y(t„ + At) + 0(At 3 ). (13.37) 

The future value Y(t + AL) can be obtained by two different approaches: 

• predictor-corrector method 

Since/(T(r + ^), t„ + ^) is multiplied with At, it is sufficient to use an approxi¬ 
mation with lower error order. Even the explicit Euler step is sufficient. Together the 
following algorithm results: 


Fig. 13.5 Improved Euler 
method 
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Fig. 13.6 Improved polygon 
(or Heun) method 



predictor step: Y (p) = Y{t „) + ff(Y(t n ), t„) 

corrector step: Y(t„ + At) = Y(t„) + Atf(Yt„ + y). 

• averaging (Heun method) 

The average of f(Y{t n ), t„) and f(Y(t n + At), t + At) is another approximation to 
the midpoint value of comparable quality (Fig. 13.6). 

Expansion around t n + At /2 gives 

2 ( f{Y(t„), t„) +f(Y(t n + At), t + At)) 

= /(^(?« +y),?„ +y)+0(^ 2) . (13.39) 

Inserting the average in (13.36) gives the following algorithm, which is also known 
as improved polygon method and corresponds to the trapezoidal rule for the integral 
(4.13) or to a combination of explicit and implicit Euler step: 


Y(t n + At) = Y(t a ) + — (f(Y(t n ), t n ) +f(Y(t n + At), t + At)). (13.40) 


In the special case of a linear function f{Y{t), t) = F Y(t) (for instance rotational 
motion or diffusion) this can be solved formally by 

Y(tn + At) = (l - ^1 + yfj Y(t n ). (13.41) 

Numerically it is not necessary to perform the matrix inversion. Instead a linear 
system of equations is solved: 

(i - y f ) Y(t " + At) = ( ! + y f ) Y(tn) - 


(13.42) 
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In certain cases the Heun method conserves the norm of the state vector, for instance 
if F has only imaginary eigenvalues (as for the 1 -dimensional Schroedinger equation, 
see p. 526). 

In the general case a predictor step has to be made to estimate the state vector at 
t„ + At before the Heun expression (13.40) can be evaluated: 

Y (p) = Y (f„) + Atf(Y(t n ), tn). (13.43) 


13.6 Taylor Series Methods 


Higher order methods can be obtained from a Taylor series expansion 

At 1 df(Y(t n ), t „) 

Y(t„ + At) = Y(t n ) + Atf (Y(t n ), t„) + — J \ n> + • • • . (13.44) 

2 dr 


The total time derivative can be expressed as 


7 

dr 


df d Y df , ■ 

— -1 - — =f'f + f 

dY dr dt J J J 


(13.45) 


where the partial derivatives have been abbreviated in the usual way by = / and 
jjy = /'. Higher derivatives are given by 


d7 

dr 2 


=/ 7 2 +f' 2 f + 2/7 +f 


% = % +f"f + 3 / 7 2 +//' + 3 / 7 / 


+ 3 f + 4 fff 2 + 5 fff +/7 +/7- 


(13.46) 


(13.47) 


13.6.1 Nordsieck Predictor-Corrector Method 

Nordsieck [151] determines an interpolating polynomial of degree m. As variables 
he uses the 0th to mth derivatives 2 evaluated at the current time f, for instance for 
m — 5 he uses the variables 


2 In fact the derivatives of the interpolating polynomial which exist even if higher derivatives off 
do not exist. 
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Y(t) 


git ) = 


d 
d t 


Y (t) 


a(t) 


b(t) 
c(t ) 


dit) 


At i 

d 2 

_V 

2 dr 2 

At 2 

d 3 

~ 6 ~ 

dr 3 

Af 

d 4 

"24" 

dr 4 

At 4 

d 5 

120 

dr 5 
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(13.48) 

(13.49) 

(13.50) 

(13.51) 

(13.52) 

(13.53) 


Taylor expansion gives approximate values at t + At 

Y(t 4- At) = Y(t) + At [git) + n(r) + b{t) + c(r) 4- d(t) + e(?)] 

= Y p (t+At) + e(t)At (13.54) 

g(t + At) = g(t)+2a(t) + 3b(t) + 4c(t)+5d(t) + 6e(t) = gP (t +AT) + 6e(t) (13.55) 

a{t + At) = a(t) + 3 b(t) + 6 c(t) + 10 d(t) + 15 e(t) = aP(f + zlt) + 15e(t) (13.56) 

b(t + At) = b(t) + 4 c(t) + 10rf(f) + 20 e(t) = Z/(? + At) + 20e(f) (13.57) 


c(f + zlt) = c(t) + 5 d{t) + 15 e(t) = c p u + zlt) + 15e(r) (13.58) 

d(t + At) = rf(t) + 6e(t) = d p (t + Z\r) + 6e(t) (13.59) 

where the next term of the Taylor series e{t) = /] f ', j',, Y(t) has been introduced as 
an approximation to the truncation error of the predicted values Y p , <f, etc. It can be 
estimated from the second equation 

e = \ [ f(Y p (t + At), t + At)- cf{t + At)] = l -5f. 


(13.60) 
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This predictor-corrector method turns out to be rather unstable. However, stability 
can be achieved by slightly modifying the coefficients of the corrector step. Nordsieck 
suggested to use 


y j 

Y (t + At) = Y p (t + At) + —5f 

Zoo 

(13.61) 

25 

a{t + At) — a p {t + At) + —5/ 

(13.62) 

35 

bit + At) = If it + At) + —Sf 

(13.63) 

c)t + At) — d’it + At) + —<5/ 

(13.64) 

dit + At) — d l \t + At) + ■ 

(13.65) 


13.6.2 Gear Predictor- Corrector Methods 

Gear [152] designed special methods for molecular dynamics simulations (Chap. 15) 
where Newton’s law (13.15) has to be solved numerically. He uses again a truncated 
Taylor expansion for the predictor step 








At 2 


At 2 

At 4 


r it 

+ 

At) = 

= r it) 

+ \it)At 

+ a it) 

2 

+ a(r) 

6 

+ a( t)—+ ••■ 

(13.66) 







At 2 


At 2 



sit 

+ 

At) = 

- v(0 

T a it) At 

+ a it) 

2 

+ a(f) 

6 

+ ■■■ 

(13.67) 







At 2 





a it 

+ 

At) - 

= a it) 

+ a it) At 

+ a(t) 

~Y 




(13.68) 

a it 

+ 

At) - 

= a(r) 

+ ait) At 

H- 





(13.69) 


to calculate new coordinates etc. r^ +1 , v^ +1 , a^ +1 ... (Fig. 13.7). The difference 
between the predicted acceleration and that calculated using the predicted coor¬ 
dinates 


<5a„+i = a(r'' + ,, t + At) - sf n+l 


(13.70) 


13.6 Taylor Series Methods 
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Fig. 13.7 (Gear Predictor 
Corrector Method) The 
difference between predicted 
acceleration aP and 
acceleration calculated for 
the predicted coordinates 
a(r p ) is used as a measure of 
the error to estimate the 
correction 5r 



is then used as a measure of the error to correct the predicted values according to 
r„+i = r^ +1 + c\5a„+i (13.71) 

v„+i = v£ +1 + c 2 S a„+i (13.72) 


The coefficients c, were determined to optimize stability and accuracy. For instance 
the fourth order Gear corrector reads 


_ ,, , At2 x 

r n+l — *n+\ H" Y2 ^ a «+l 


P , 5 At 

v„+t = < +1 + — 5a„+i 


a«+i — + ha yJ 4-|. 


(13.73) 

(13.74) 

(13.75) 


Gear methods are generally not time reversible and show systematic energy drifts. 
A reversible symplectic predictor-corrector method has been presented recently by 
Martyna and Tuckerman [153]. 


13.7 Runge-Kutta Methods 

If higher derivatives are not so easily available, they can be approximated by numer¬ 
ical differences. / is evaluated at several trial points and the results are combined to 
reproduce the Taylor series as close as possible [154]. 
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13.7.1 Second Order Runge-Kutta Method 

Let us begin with two function values. As common in the literature we will denote 


the function values as K \, K 2 , .... From the gradient at time t„ 

K\ —fn =f(Y(t n ), t n ) (13.76) 

we estimate the state vector at time t„ + At as 

Y(t„ + At) « AtKi. (13.77) 

The gradient at time t n + At is approximately 

K 2 =f(Y(t„) + AtK u t n + At) (13.78) 

which has the Taylor series expansion 

K 2 =fn + (fn +f’Jn)At + • ■ • (13.79) 

and application of the trapezoidal rule (4.13) gives the 2nd order Runge-Kutta method 

Y n+ i = Y n + =-(Ki + K 2 ) (13.80) 

which in fact coincides with the improved Euler or Heun method. Taylor series 


expansion shows how the combination of K\ and K 2 leads to an expression of higher 
error order: 


Y„+l — Y n + — (f„ +f n + (f„ + f'fn)At + • • • ) 
d f, At 2 

= Y n +f n At+^- + --- . (13.81) 

13.7.2 Third Order Runge-Kutta Method 

The accuracy can be further improved by calculating one additional function value 
at mid-time. From (13.76) we estimate the gradient at mid-time by 

/ At At\ 

*2=/(r(0 + T *,* + T ) 

= fn + (fn +Wy + ^ + fnfn + tfn)^ +■■■■ 


(13.82) 


13.7 Runge-Kutta Methods 
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The gradient at time t n + At is then estimated as 
Ki =f(Y (f„) + At(2K 2 - Ki), t n + At) 

= fn +f„At +f;,(2K 2 - Ki)At +fn^~ 

, j. n (2K 2 — K\) 2 At 2 , o ., (2K 2 -K x )At 2 , 

' Jn ~ ' Ain + ' ' ' 


(13.83) 


Inserting the expansion (13.82) gives the leading terms 

At 2 

K 3 — fn + tin +fX)At + (2/,; 2 f n +f''f 2 +f n + 2 fjn + 2/„ 2 ) — + • • • . (13.84) 

Applying Simpson’s rule (4.14) we combine the three gradients to get the 3rd order 
Runge-Kutta method 


Yn+ 1 = T(f„) + + 4 K 2 + K 3 ) 

o 


where the Taylor series 


(13.85) 


Y n + 1 — T(t„) + — (6/, + 3(/j, A fnf'n) At 

+ tin 2 fn +/;/„ 2 + AfX + f n + 'X)At 2 + • ■/) 

= Y(t n + At) + 0(At 4 ) (13.86) 


recovers the exact Taylor series (13.44) including terms of order 0(At 3 ). 


13.7.3 Fourth Order Runge-Kutta Method 

The 4th order Runge-Kutta method (RK4) is often used because of its robustness 
and accuracy. It uses two different approximations for the midpoint 

K i =f(Y(t„),t n ) 

( K x At\ 

K 2 =f\Y(t n ) + ^-At,t n + — \ 

( K 2 At\ 

K 3 = flY(t n ) + ^At,t n + — \ 

K 4 — f(Y(t n ) + K 3 At, t n + At) 
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and Simpson’s rule (4.14) to obtain 

Yn+ 1 = Y(t n ) + ^(K { + 2K 2 + 2 Ki + K 4 ) = Y(t n + At) + 0(At 5 ). 
o 

Expansion of the Taylor series is cumbersome but with the help of an algebra program 
one can easily check that the error is of order At 5 . 


13.8 Quality Control and Adaptive Step Size Control 

For practical applications it is necessary to have an estimate for the local error and to 
adjust the step size properly. With the Runge Kutta method this can be achieved by 
a step doubling procedure. We calculate y n+2 first by two steps At and then by one 
step 2 At. This needs 11 function evaluations as compared to 8 for the smaller step 
size only (Fig. 13.8). For the 4th order method we estimate the following errors: 


A (^ 2 ) = 2 aAt 5 
A ( Y n+7 ] ) = a{2At) 5 . 


(13.87) 


(13.88) 


The local error can be estimated from 



Fig. 13.8 Step doubling 



with the fourth order 
Runge-Kutta method 


n 


n+2 



n 


n+1 


n+2 
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A 



= a At 5 


| y(At) y(2At) | 

I I n+2 1 n+2 I 

30 


The step size At can now be adjusted to keep the local error within the desired limits. 


13.9 Extrapolation Methods 

Application of the extrapolation method to calculate the integral //" +I f(t)dt produces 
very accurate results but can also be time consuming. The famous Gragg-Bulirsch- 
Stoer method [2] starts from an explicit midpoint rule with a special starting proce¬ 
dure. The interval At is divided into a sequence of N sub-steps 

At 

h = —. (13.89) 

N 

First a simple Euler step is performed 


Mo = Y(t n ) 

Ml = M 0 + hf(uo, t„) (13.90) 


and then the midpoint rule is applied repeatedly to obtain 


u J+ \ =Uj-i +2hf(uj,t n +jh) j = 1,2...A- 1. 


(13.91) 


Gragg [155] introduced a smoothing procedure to remove oscillations of the leading 
error term by defining 


1 1 1 

v i = + 2 U J + 4 “M-i- 


(13.92) 


He showed that both approximations (13.91, 13.92) have an asymptotic expansion in 
powers of h 2 and are therefore well suited for an extrapolation method. The modified 
midpoint method can be summarized as follows: 


«0 = Y(t„) 

Ml = M 0 + hf (n 0 , t n ) 

Uj+\ = Uj-i + 2hf(uj , t n +jh ) j = 1, 2, ... N - 1 

1 

Y (f„ + At) « - (u N + u N _i + hf(u N , t n + At)) . 


(13.93) 
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The number of sub-steps N is increased according to a sequence like 

N = 2, 4, 6, 8, 12, 16, 24, 32, 48, 64 ... Nj = 27V,-_ 2 Bulirsch-Stoer sequence 

(13.94) 


or 


N = 2,4, 6, 8, 10, 12 ... /V/ = 2/ Deuflhard sequence. 

After each successive N is tried, a polynomial extrapolation is attempted. This extrap¬ 
olation returns both the extrapolated values and an error estimate. If the error is still 
too large then N has to be increased further. A more detailed discussion can be found 
in [156, 157], 


13.10 Linear Multistep Methods 

All methods discussed so far evaluated one or more values of the gradient/(T(f), t) 
only within the interval t„ ■■■ t n + At. If the state vector changes sufficiently smooth 
then multistep methods can be applied. Linear multistep methods use a combination 
of function values Y„ and gradients f, from several steps 

k 

Y n +\ = (a/ Y n ~j+t + 0jfi—j+i At) + 0of n +\At (13.95) 

7=1 

where the coefficients a, 0 are determined such, that a polynomial of certain order 
r is integrated exactly. The method is explicit if 0 O = 0 and implicit otherwise. 
Multistep methods have a small local error and need fewer function evaluations. On 
the other hand, they have to be combined with other methods (like Runge-Kutta) to 
start and end properly and it can be rather complicated to change the step size during 
the calculation. Three families of linear multistep methods are commonly used: 
explicit Adams-Bashforth methods, implicit Adams-Moulton methods and backward 
differentiation formulas (also known as Gear formulas [158]). 


13.10.1 Adams-Bashforth Methods 

The explicit Adams-Bashforth method of order r uses the gradients from the last 
r — 1 steps (Fig. 13.9) to obtain the polynomial 


p(t„) —f (Y n , t n ), . . ■ pOn—r+l ) —f ( Yn , +1 , t n _ r _ (_i) 


(13.96) 
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Fig. 13.9 Adams-Bashforth 
method 



and to calculate the approximation 



which is generally a linear combination of/„ ■ ■ ■f n ~ r + 1 - For example, the Adams- 
Bashforth formulas of order 2, 3, 4 are: 

Yn +1 - Y„ = y (3 f n -fn- 1) + 0{At 3 ) 

Y „+1 - Y n = ^(23/„ - 16/„_! + 5/ m _ 2 ) + 0(Ar 4 ) 

y»+i - = ^(55/„ - 59/„_i + 37/,_ 2 - 9/„_ 3 ) + 0(At 5 ). (13.97) 


13.10.2 Adams-Moulton Methods 

The implicit Adams-Moulton method also uses the yet not known value Y ll+ \ 
(Fig. 13.10) to obtain the polynomial 

P0n+l') —/i+li • • • POn—r+f) — fn—r+2' (13.98) 

The corresponding Adams-Moulton formulas of order 2 to 4 are: 

Yn +1 - Y n = ~(f n+1 +/„) + 0(At 3 ) 

Yn+1 - Y„ = ^(5/„+l + 8 fn -fn- 1) + 0(Ar 4 ) (13.99) 

F/,+1 — T„ = — (9/„ + i + I9f, — 5/,_i T fn—i) + 0{At 5 ). 


(13.100) 






308 


13 Equations of Motion 


Fig. 13.10 Adams-Moulton 
method 



-- t 

1 n-1 l n l n+1 


13.10.3 Backward Differentiation (Gear) Methods 

Gear methods [158] are implicit and usually combined with a modified Newton 
method. They make use of previous function values Y n , F„_i... and the gradient 
/„+i at time t + At. Only methods of order r < 6 are stable and useful. The general 
formula (13.95) is 

r 

Y n + 1 =Y J a i Y n-j+l+Mn+lAt. (13.101) 

j= 1 

For r — 1 this becomes 

Y n+l = ai Y n + PoftAt (13.102) 

and all linear polynomials 

P=P0+Pi(t-t n ), ^-=p\ (13.103) 

at 

are integrated exactly if 

Po +PiAt = aip 0 + fiopi (13.104) 

which is the case for 

a 1 = l, Po — At. (13.105) 

Hence the first order Gear method is 


Yn +1 =Y„ +f„ + \At + 0(At 2 ) 


(13.106) 
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which coincides with the implicit Euler method. The higher order stable Gear meth¬ 
ods are given by 


r = 2: Y n+1 = ^Y n - ^Y^+ ^f n+1 At + 0(At 3 ) (13.107) 

r = 3 : Y n+l = ^Y n -^Y n _ x + ^Y n _ 2 + ^f n+l At + 0{At A ) (13.108) 

48 36 16 3 12 , 

r = 4 : Y n+1 = — Y„ — —Y n _ 1 + — Y„_ 2 - — Y„_ 3 + —f n+l At + 0(At 5 ) 


300 300 200 75 

r = 5 : T„+i = - Y„ -T„_i +-F„_ 2 -T„_ 3 

+ 137 137 137 137 


12 


60 


+ ~yy-j Y n—4 + J^jfn+lAt + 0(At ) 


120 150 400 75 

r = 6: T„+i = - Y n -T„_j +-F„_ 2 -T„_ 3 

+ 49 49 " 147 49 

24 10 20 7 

+ 49 y "“ 4 “ T47 F "- 5 + 49 fn+lAt + ° (At 


(13.109) 


(13.110) 


(13.111) 


This class of algorithms is useful also for stiff problems (differential equations with 
strongly varying eigenvalues). 


13.10.4 Predictor- Corrector Methods 

The Adams-Bashforth-Moulton method combines the explicit method as a predictor 
step to calculate an estimate yjj +1 with a corrector step using the implicit method of 
same order. The general class of linear multistep predictor corrector methods [159] 
uses a predictor step 

C = Z ( a< i P>Y '‘-.^ + P^fn-j+iAt) (13.112) 

7=1 

which is corrected using the formula 

k 

Y n+l = Z ( a f Y «-j+ 1 + Pj C) fn~j+l At ) + /Wi+V t n+1 )At 


(13.113) 
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and further iterations 


v (m+ 1 ) _ v ( m ) 
I n +1 _ *11+1 


A) [f(Y% 7 1} , t n+ 1) -/(T„% f„+l)] 4r m = 1 . . . M - 1 


(13.114) 



(13.115) 


The coefficients a, (3 have to be determined to optimize accuracy and stability. 


13.11 Verlet Methods 

For classical molecular dynamics simulations it is necessary to calculate very long 
trajectories. Here a family of symplectic methods often is used which conserve the 
phase space volume [160-165], The equations of motion of a classical interacting 
N-body system are 

niiXi = Fi (13.116) 

where the force acting on atom i can be calculated once a specific force field is 
chosen. Let us write these equations as a system of first order differential equations 



(13.117) 


where x(t) and v(f) are functions of time and the forces ma(x(t)) are functions of 
the time dependent coordinates. 


13.11.1 Liouville Equation 

We rewrite (13.117) as 

OHO) 

where the Liouville operator C acts on the vector containing all coordinates and 
velocities: 
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The Liouville equation (13.118) can be formally solved by 

\ v (0) / 


For a better understanding let us evaluate the first members of the Taylor series of 
the exponential: 


C 




(13.121) 


C 2 


C? 



+ 4)(aw)-(v»a) 

+ 4 ) ( 4 .)-(*&«.) 


(13.122) 

(13.123) 


But since 


d 

df 


a(x(f)) = v—a 
ox 


(13.124) 


df 2 


a(x(/)) 



d d d 

a^-a + vv——a 
ox ox ox 


we recover 


(13.125) 


^1 + tC, + 




\ / x \ _/ x + vf + 5 f2a + S r3a + -"\ 
/\ v / \v + ar+ 4 2 a+ |r 3 a + ---/ 


(13.126) 


13.11.2 Split Operator Approximation 

We introduce a small time step At = t/N and write 

e Ct = (e CA, ) N . (13.127) 

For the small time step At the split-operator approximation can be used which approx¬ 
imately factorizes the exponential operator. For example, write the Liouville operator 
as the sum of two terms 
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„ d „ d 

Ca = v ^~~ Cb = a ^~~ 
ox o\ 

and make the approximation 

0 CAt _ C\At CbA t 


e — e 




(13.128) 


Each of the two factors simply shifts positions or velocities 


„^ A At | 


(0-(' + D ^'(v) = (v + aa,) 


(13.129) 


since these two steps correspond to either motion with constant velocities or constant 
coordinates and forces. 


13.11.3 Position Verlet Method 

Often the following approximation is used which is symmetrical in time 

e CA ‘ = e C-AAtl1 e C B At e C, A Atl2 H -_ (13.130) 

The corresponding algorithm is the so called position Verlet method (Fig. 13.11): 
At 

x„+y 2 = x„ 4~ v„— (13.131) 

v„+i = v„ + a„+i /2 At = \{t n + At) + 0(At 3 ) (13.132) 

x„+i = x„+ii /2 +v B+1 y = X„+ V ” + 9 V " +I At = x(f n + z\f) + c>(2\f 3 ). (13.133) 


Fig. 13.11 (Position Verlet 
method) The exact 
integration path is 
approximated by two 
half-steps with constant 
velocities and one step with 
constant coordinates 


V 



X 
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Fig. 13.12 (Velocity Verlet 
method) The exact 
integration path is 
approximated by two 
half-steps with constant 
coordinates and one step 
with constant velocities 


V 



13.11.4 Velocity Verlet Method 

If we exchange operators A and B we have 

e CAt = e CBAt/2 e C A Ar e C B At/2 ^- (13.134) 

which produces the velocity Verlet algorithm (Fig. 13.12): 

At 

v„+i /2 = v„ + a„— (13.135) 

At 2 , 

x„+i = x n +\ n+ y 2 At = x n + \„At + a n — = x(t„ + At) + O(Ar) (13.136) 

~\~ 3 / 2+1 T, 

v„+i = v„+i /2 + a„+i — = v„ H--- At = v(f„ + At) + 0(At ). 

(13.137) 


13.11.5 Stoermer-Verlet Method 

The velocity Verlet method is equivalent to Stoermer’s version [166] of the Verlet 
method which is a two step method given by 


V /+1 — 2x„ x ; ,_ j T a„ At 


(13.138) 


Xn+l X /; _ | 

2 At ' 


(13.139) 


To show the equivalence we add two consecutive position vectors 


V +2 T ^-ii 1 — 2x„ + i T 2x„ x n x„ _i T (a n +i T a„ ) At 


(13.140) 
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which simplifies to 


X//+2 X n (X/i+1 X M ) — 1 “f~ tl ;; ) At~ . 


(13.141) 


This can be expressed as the difference of two consecutive velocities: 


2(v„+i v n ) — T a 


(13.142) 


Now we substitute 


Xh— i —x H ^_i 2.\ n At 


(13.143) 


to get 


x^; 11 — 2x„ x „ + 1 T 2 YuZ\ f -(- a„z\r 


(13.144) 


which simplifies to 

x„+i = x„ + v„zlf + (13.145) 

Thus the equations of the velocity Verlet algorithm have been recovered. However, 
since the Verlet method is a 2-step method, the choice of initial values is important. 
The Stoermer-Verlet method starts from two coordinate sets x 0 , X\. The first step is 

xt = 2xi — xo + ci\At 2 (13.146) 


Vl 


X 2 -x 0 
2 At 


xi - x 0 
At 



(13.147) 


The velocity Verlet method, on the other hand, starts from one set of coordinates and 
velocities Xj, Vi. Here the first step is 


X 2 = X! + vi At + ai 


At 2 

~Y 


(13.148) 


V2 = Vi + 


ai +a 2 
2 


At. 


(13.149) 


The two methods give the same resulting trajectory if we choose 
Si\ j 

x 0 = xi - Vi At + —A t . 


(13.150) 


If, on the other hand, xo is known with higher precision, the local error order of 
Stoermer’s algorithm changes as can be seen from addition of the two Taylor series 
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(13.151) 



(13.152) 


which gives 


x(f„ + At) = 2x(t„) — x(t n — At) + a „At 2 + 0{At 4 ) 


(13.153) 


x(t„ + At) - x(r„ - At) 
2 At 


= v„ + 0{Ar). 


(13.154) 


13.11.6 Error Accumulation for the Stoermer-Verlet Method 

Equation(13.153) gives only the local error of one single step. Assume the start 
values xo and Xi are exact. The next value X 2 has an error with the leading term 
Ax 2 = aAt 4 . If the trajectory is sufficiently smooth and the time step not too large 
the coefficient a will vary only slowly and the error of the next few iterations is given 
by 

Ax 3 = 2 Ax 2 — Ax i = 2 aAt 4 
Ax 4 = 2Ax 3 — Ax 2 = 3aAt 4 

Ax„ + \=naAt 4 . (13.155) 

This shows that the effective error order of the Stoermer-Verlet method is only 0(At 3 ) 
similar to the velocity Verlet method. 


13.11.7 Beeman’s Method 

Beeman and Schofield [167, 168] introduced a method which is very similar to the 
Stoermer-Verlet method but calculates the velocities with higher precision. This is 
important if, for instance, the kinetic energy has to be calculated. Starting from the 
Taylor series 


At 2 . At 2 .. At 4 

Xh+i = x„ -f- v„ At -{- ti/j — -j- a,) — -[- a,,—— + 
2 o 24 


(13.156) 
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the derivative of the acceleration is approximated by a backward difference 


At 2 a„ — a„_i At 3 d 

x„ +1 = x„ + v„At + a„ ——I---— + O(Ar) 

l At o 

4a,, — a w _i 9 /t 

= x„ + v„z\r H- Ar + 0(At 4 ). 


(13.157) 


This equation can be used as an explicit step to update the coordinates or as a predictor 
step in combination with the implicit corrector step 


At 2 a„+i — a„ At 3 . 

x„ +1 = x„ + \„At + a„ ——I---— + 0(At 4 ) 

z At o 

= x„ + \ n At + a " +1 + ~ a ” At 2 + 0(At 4 ) 

6 


(13.158) 


which can be applied repeatedly (usually two iterations are sufficient). Similarly, the 
Taylor series of the velocity is approximated by 


. At 2 .. At 3 
v„+i = v„ + a n At + a„ ——I- a„ —— 
z o 


v„+a n At + 
a«+i “I - a,, 


a, t+ i a w 
At 


\ At 2 
O(At)) — 


= v„ 


z\t + 0 (z\ri). 


(13.159) 


Inserting the velocity from (13.158) we obtain the corrector step for the velocity 


v n+1 = X,i+1 „, X ” - a » +1 + 2a » At + a “+'+ an At + 0(At 3 ) 


At 6 

x„ 2a„+i T a„ 


z\t + C)(zit 3 ). 


6 

In combination with (13.157) this can be replaced by 


(13.160) 


4a„ — a„_i 2a„ + i + a„ 3 

v„+i — v„ H--- At H- - - At + 0(At ) 

6 6 

= V„ + 2a »+l + 5a » ~ a >-l Af + Q{At 


(13.161) 


Together, (13.157) and (13.161) provide an explicit method which is usually 
understood as Beeman’s method. Inserting the velocity (13.160) from the previous 
step 


ttn x H _i 2a H T a;j_i j 

v„ = ---+--- At + 0(At 3 ) 

At 6 


(13.162) 
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into (13.157) gives 

x„ +1 = 2x„ — x„_! + a n At 2 + 0(At A ) (13.163) 

which coincides with the Stoermer-Verlet method (13.138). We conclude that Bee- 
man’s method should produce the same trajectory as the Stoermer-Verlet method if 
numerical errors can be neglected and comparable initial values are used. In fact, the 
Stoermer-Verlet method may suffer from numerical extinction and Beeman’s method 
provides a numerically more favorable alternative. 


13.11.8 The Leapfrog Method 

Closely related to the Verlet methods is the so called leapfrog method [165]. It uses 
the simple decomposition 

e CAt ^ e c A At e c B A, (13.164) 

but introduces two different time grids for coordinates and velocities which are shifted 
by At/2 (Fig. 13.13). 

The leapfrog algorithm is given by 

v„+ 1/2 = v,j— 1 /2 + a „At (13.165) 

x„+i = x„ + v„+i /2 At. (13.166) 

Due to the shifted arguments the order of the method is increased as can be seen 
from the Taylor series: 

x(r„) + C v(f„) + y a(r„) + • • • j At = x(f B + At) + 0(At 3 ) (13.167) 


Fig. 13.13 (Leapfrog 
method) The exact 
integration path is 
approximated by one step 
with constant coordinates 
and one step with constant 
velocities. Two different 
grids are used for 
coordinates and velocities 
which are shifted by At /2 


V f 



X 


X 
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v - v p - p'j = a (t„)At + 0(At 3 ). (13.168) 

One disadvantage of the leapfrog method is that some additional effort is necessary 
if the velocities are needed. The simple expression 

Pjj + 0(At 2 ) (13.169) 

is of lower error order than (13.168). 


v(f„) = - 


= K v ('"* f) +y (‘" + 


Problems 


Problem 13.1 Circular Orbits 


In this computer experiment we consider a mass point moving in a central field. The 
equation of motion can be written as the following system of first order equations: 


/ x\ 

/ 

y 


V x 

U’v/ 

V 


0 

0 


U 2 +v 2 ) 3 / 2 

0 


0 1 0 \ 

0 0 1 

0 00 

~(x 2 +v 2 ) 3/2 00 y 


( x\ 

y 


V x 

w 


(13.170) 


For initial values 


x\ 


/ 1 \ 

y 


0 

Vx 


0 

w 


w 


(13.171) 


the exact solution is given by 


x = cos t y — sin t. 


(13.172) 


The following methods are used to calculate the position x(t), y(t ) and the energy 

E, ot = E kin + Epg, = ]-(v 2 x + v 2 ) - . 1 . (13.173) 

1 v x ~ + y 


• The explicit Euler method (13.3) 
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x(j n +\) — x(t n ) T V x (t n )At 
y(t„+ 1 ) = y(t n ) + v y (t„)At 

Vx(tn+ 1) — Pjc (fn) R(t ',') 3 ^ 

Vy(tn+\ ) — Vy(tn) R(l, ) 3 ^' 

• The 2nd order Runge-Kutta method (13.7.1) 
which consists of the predictor step 

At 

x(t„ + At/2) = x(t n ) + — v x (t„) 

At 

y(t„ + At/ 2) = y(f„) + —v y (t n ) 

At x{t n ) 

v x (t n + zit/2) = v x (t n ) - — ——r 

Z /v j 

z\r v(f„) 

Vy(t n + At/2) = v y (t n )--—f^ 

^ K\*n) 


and the corrector step 


x(t n+ 1 ) = x(t n ) + At v x (t n + At/2) 


y(r„+i) = y(t„) + At v y (t„ + At/2) 


v x (tn-\- \) — r. (if/) 


x(t„ + At/2) 
R\t n +At/2) 


Vy(/n+l) — Vy(t/i) At 


y(tn + At/2) 
RHt n +At/2)' 


• The fourth order Runge-Kutta method (13.7.3) 

• The Verlet method (13.11.5) 


x(t n+ i) = x(t n ) + (x(t„) - x(r„_i)) - At 


x(t„) 

RHtn) 


y(t„+ 1 ) = yd,,) + (y(t„) - y{t n -\)) - At 


y(t n ) 

R 3 (t n ) 


C’ (jn ) — 
Vy (fn) = 


x(t n+ i) - x(t n -i) x(t n )-x(t n - 1 ) At x(t„) 


2At At 2 R 3 (t n ) 

y{t n+l ) -y(t„_i) y(t n ) - y(t n _i) At y(t n ) 


2 At 


At 


2 R*a n y 


(13.174) 

(13.175) 

(13.176) 

(13.177) 

(13.178) 

(13.179) 

(13.180) 

(13.181) 

(13.182) 

(13.183) 

(13.184) 

(13.185) 

(13.186) 
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To start the Verlet method we need additional coordinates at time —At which can be 
chosen from the exact solution or from the approximation 


x(t- 1 ) = x(f 0 ) - At v x (t 0 ) - 


At 2 x(to) 


y(t- 1) = y(t 0 ) - At Vy(t 0 ) - 


At 2 y(t 0 ) 

2 R 3 (t 0 y 


• The leapfrog method (13.11.8) 


xitn+ 1) —x(f n ) T V x {t 11 ^/ 2 )At 


(13.187) 

(13.188) 


(13.189) 


y(t n+ 1 ) = y(t„) + v y {t n+ y 2 )At 


(13.190) 


x(t n ) 

Vx(fn+yi) — Tr(6i— 1 / 2 ) — ———r At 
K\tn) 

y(t n ) 

Vy{tn-\-i/i) = Vyifn— 1 / 2 ) TTZ 

K\}n) 

where the velocity at time f„ is calculated from 


t'x (tn) — Vx(t n + 1/2) 


At x(t n+ 1 ) 

2 7? 3 (f„ + i) 


t>V (til ) - tt-y (t n -j-l/ 2 ) 


yfa+i) 

2 7? 3 (r„ + i) 


(13.191) 

(13.192) 


(13.193) 

(13.194) 


To start the leapfrog method we need the velocity at time f_y 2 which can be taken 
from the exact solution or from 

At x(fo) 

vAt- Vi) = Wx«b) - (13.195) 

At y(t 0 ) 

Vy (t_ 1/2 ) = Vy(t 0 )- Y ^ ) . (13.196) 

Compare the conservation of energy for the different methods as a function of the 
time step At. Study the influence of the initial values for leapfrog and Verlet methods. 

Problem 13.2 N-body System 

In this computer experiment we simulate the motion of three mass points under the 
influence of gravity. Initial coordinates and velocities as well as the masses can be 
varied. The equations of motion are solved with the 4th order Runge-Kutta method 
with quality control for different step sizes. The local integration error is estimated 
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using the step doubling method. Try to simulate a planet with a moon moving round 
a sun! 

Problem 13.3 Adams-Bashforth Method 

In this computer experiment we simulate a circular orbit with the Adams-Bashforth 
method of order 2 ... 7. The absolute error at time T 

A(T) = \x(T) -cos(T)| + | y(t) - sin(T)| + \v x (T) + sin(r)| + ^(T) - cos(T)| 

(13.197) 


is shown as a function of the time step At in a log-log plot. From the slope 


^QogioC^)) 

d(logi 0 (Af)) 


(13.198) 


the leading error order .v can be determined. For very small step sizes rounding errors 
become dominating which leads to an increase A ~ (At) . 

Determine maximum precision and optimal step size for different orders of the 
method. Compare with the explicit Euler method. 



Part II 

Simulation of Classical 
and Quantum Systems 


Chapter 14 

Rotational Motion 


An asymmetric top under the influence of time dependent external forces is a rather 
complicated subject in mechanics. Efficient methods to describe the rotational motion 
are important as well in astrophysics as in molecular physics. The orientation of a 
rigid body relative to the laboratory system can be described by a 3x3 matrix. 
Instead of solving nine equations for all its components, the rotation matrix can 
be parametrized by the four real components of a quaternion. Euler angles use the 
minimum necessary number of three parameters but have numerical disadvantages. 
Care has to be taken to conserve the orthogonality of the rotation matrix. Omelyan's 
implicit quaternion method is very efficient and conserves orthogonality exactly. 
In computer experiments we compare different explicit and implicit methods for a 
free rotor, we simulate a rotor in an external field and the collision of two rotating 
molecules. 


14.1 Transformation to a Body Fixed Coordinate System 


Let us define a rigid body as a set of mass points m, with fixed relative orientation 
(described by distances and angles). 

The position of m, in the laboratory coordinate system CS will be denoted by rp 
The position of the center of mass (COM) of the rigid body is 


R = 


X,' 




msTj 


(14.1) 


and the position of m, within the COM coordinate system CS, (Fig. 14.1) is p, : 

r, = R + Pi- (14.2) 

Let us define a body fixed coordinate system CS c b, where the position p ib of m, is 
time independent 4 p ih — (). p i and p ih are connected by a linear vector function 
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Fig. 14.1 (Coordinate 
systems) Three coordinate 
systems will be used: The 
laboratory system CS, the 
center of mass system CS c 
and the body fixed system 

CS cb 


Pi = A P,b 

where A is a 3 x 3 matrix 

( fl ll «12 ^13 
a 2l «22 «23 

«31 <232 «33 




(14.3) 


(14.4) 


14.2 Properties of the Rotation Matrix 

Rotation conserves the length of p 1 : 

P T P = (Apf (Ap) = p T A T Ap. (14.5) 

Consider the matrix 

M — A t A — 1 (14.6) 

for which 

p T Mp — 0 (14.7) 


holds for all vectors p. Let us choose the unit vector in x-direction: p = 
we have 



. Then 


1 p T p denotes the scalar product of two vectors whereas pp T is the outer or matrix product. 
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(M u M 12 M 13 \ / 1 \ 

0= (1 0 0) ( M 2l M 22 M 23 ) 0 I = Mu. (14.8) 

\M 31 m 32 m 33 ) \o) 


Similarly by choosing a unit vector in y or z direction we find M 22 = M 33 = 0. 

1 
1 
0 

(M n M 12 M 13 \ / 1 \ 

0= (1 1 0) I M 21 M 22 M 23 I I 1 I 

\m 31 m 32 m 33 J \o/ 

/Mp + Mu \ 

= (1 1 0) j M 2 1 + Mil I = Mu + M 22 + Mi 2 + M 2 i. (14.9) 

\M 3 i + M 32 y 


Now choose p = 


l \ 

Since the diagonal elements vanish we have Mp = —M 2 p With p = I 0 1 , p = 

V 1 / 

'°\ 

1 I we find Mp = —Mu and Mi 3 = —Mu, hence M is skew symmetric and has 

V 

three independent components 


0 M\2 Mu 

M = — M 1 = [ — M\2 0 M 23 

—Mu —m 23 0 


(14.10) 


Inserting (14.6) we have 

(A t A - 1) = —{A 7 A - \) T = -(A t A - 1) (14.11) 

which shows that A 7 A = 1 or equivalently A 1 = A -1 . Hence (det(A)) 2 = 1 and A 
is an orthogonal matrix. For a pure rotation without reflection only det(A) = +1 is 
possible. 

From 


ri — R + Ap ib 


we calculate the velocity 


dri 

d t 


dR 

~d7 


dA 

dr 


Pib + A 


d Pib 

dr 


(14.12) 


(14.13) 


328 


14 Rotational Motion 


but since p ih is constant by definition, the last summand vanishes 

r, = R + Ap ih = R + AA~’p, (14.14) 

and in the center of mass system we have 

^-p i =AA- x p i = Wp i (14.15) 

at 

with the matrix 

W=AA~ 1 . (14.16) 

14.3 Properties of W, Connection with the Vector 
of Angular Velocity 

Since rotation does not change the length of p h we have 

0 = 0 = = Pi(Wpi) (14.17) 

dr dr 

or in matrix notation 

0 = pjwpi- (14.18) 

This holds for arbitrary p r Hence W is skew symmetric and has three independent 
components 


/ 0 Wn Wn\ 

W = I -W n 0 W23 ) - (14.19) 

\-W13 - W23 0 / 

Now consider an infinitesimal rotation by the angle dp (Fig. 14.2). 
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Then we have (the index i is suppressed) 

, / 0 Wn Wi 3 \ / Pi \ / W 12 P 2 + W 12 P 2 \ 

dp = —dt = j -Wn 0 W 22 J { p 2 J dt = I -WnP\ + W 23 P 3 ) dt 

\ — W \2 — W22, 0 J \P 3 / \—Wi 3 pi — W23P2 J 

(14.20) 

which can be written as a cross product: 

dp — dip x p (14.21) 

with 


/ — IT 23 dt \ 

dp = I W n dt J . (14.22) 

\-Wndt) 

But this can be expressed in terms of the angular velocity u> as 

dp = ujdt (14.23) 

and finally we have 


dp — ivdt = 



( 0 —103 to 2 \ 

LO3 0 —LO\ j 
-002 Wt 0 J 


and the more common form of the equation of motion 
d 

— p= Wp= u x p. 


Example .-Rotation Around the z-axis 

For constant angular velocity to the equation of motion 


— p= Wp 
at 


has the formal solution 


P = e W ‘ p(0) = A{t) p(0). 


(14.24) 


(14.25) 


(14.26) 


(14.27) 


The angular velocity vector for rotation around the z-axis is 
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LJ — 


( : 

\U>3 


and 


/0 — to 30\ 

W = [ w 3 0 0 j 

\ 0 0 0 / 


(14.28) 


(14.29) 


Higher powers of W can be easily calculated since 

!~U)\ 0 0 \ 

W 2 = l 0 -w\ 0 J (14.30) 

\ 0 0 0 / 


0 —W 3 O' 

= -w 3 7 ( w 3 0 0 

0 0 0 , 


(14.31) 


etc., and the rotation matrix is obtained from the Taylor series 
A(t ) = e Wf = 1 + 1 Vt+- W 2 t 2 + - W 3 t 3 + ■ ■ ■ 


Loh 2 0 0 ^ 


= 1+1 0 U)\t 2 Oll- 

VO 0 0 , 


( cos ( 013 1 ) — sin(cJ 3 f) 
sin(aJ 3 t) cos (u)$t) 


( 1 \ 

V 2 + ^ + "7 


0 — ui^t O' 1 
jj^t 0 0 

0 0 0 , 


+ |at 3 f 0 0 I (l - ) 


(14.32) 


14.4 Transformation Properties of the Angular Velocity 


Now imagine we are sitting on the rigid body and observe a mass point moving 
outside. Its position in the laboratory system is rj. In the body fixed system we 
observe it at 

p li =A- 1 (n-R) (14.33) 


and its velocity in the body fixed system is 
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dA _1 

p lb =A~ 1 ( ri - R) + —(rt - R). 

at 

The time derivative of the inverse matrix follows from 

d , , . dA" 1 

0 = —(A- 1 A) = A- 1 A + - A 

d t d t 

dA- 1 , . , , 

df 

and hence 
dA -1 

——(rt — R) = —A _1 VP( ri — R). 
df 


(14.34) 


(14.35) 

(14.36) 


(14.37) 


Now we rewrite this using the angular velocity as it is observed in the body fixed 
system 


-A ‘W(it — R) = -W h A [ (ri - R) = -W h p u , — -uj b x p lb (14.38) 


where W transforms as like a rank—2 tensor 
W b = A _ 1 WA. 


(14.39) 


From this equation the transformation properties of u> can be derived. We consider 
only rotation around one axis explicitly, since a general rotation matrix can always 
be written as a product of three rotations around different axes. For instance, rotation 
around the z-axis gives: 


/ 

0 - 

■UJb3 

Ub2 \ 



Ub3 

0 

“Hi I 

= 

V 

-UJ b 2 U>bl 

0 / 


COS cp 

sin p 

°\ 

/ 0 

—UJ 3 

— sin p cos p 

0 

I w 3 

0 

0 

0 

1/ 

\-U>2 

U>1 



cos <p — sin ip 0 ' 
sin ip cos ip 0 

0 0 1 


0 —W 3 ui 2 cos p — u> 1 sin <p 

W 3 0 — (lu\ cos ip + u>2 sin <p) 

— {u)2 cos <p — lu\ sirup) u>\ cos <p + u>2 sin ip 0 


(14.40) 


which shows that 
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( ujib\ / cossin 0 \ jto\\ 

u>2 b ] = I — sin<pcos<pO J I u>2 1 =A~ 1 uj ( 14 . 41 ) 

U3b) V 0 0 


i.e. transforms like a vector under rotations. However, there is a subtle difference 
considering general coordinate transformations involving reflections. For example, 
under reflection at the xy-plane W is transformed according to 


/ 1 0 0 \ / 0 — U>3 UJ2 \ 

w b = I 0 1 0 I j w 3 0 - 0,1 
yOO — ij y — u>2 cut 0 J 

( 0 — OJ 3 —U >2 

LO 3 0 U>1 

U>2 —UJ1 0 



and the transformed angular velocity vector is 

/uib\ /I 0 0 \ (u >A 

W 2i =- 01 0 ) l U2 . 

\u3bj \°0 


(14.42) 


(14.43) 


This is characteristic of a so called axial or pseudo-vector. Under a general coordinate 
transformation it transforms as 


uj b = det(A)Atu. 


(14.44) 


14.5 Momentum and Angular Momentum 

The total momentum is 

P = niiTj = m,R = MR (14.45) 

i i 


since by definition we have ]T ( . m,p, = 0 . 

The total angular momentum can be decomposed into the contribution of the 
center of mass motion and the contribution relative to the center of mass 

L = y x f, = MR x R + ^ m,p, x p, = L com + L int . (14.46) 

i i 


The second contribution is 
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L int = y'miPi x (uj x = Y m < (yjp] - pi(piU)) . (14.47) 

i i 

This is a linear vector function of uj, which can be expressed simpler by introducing 
the tensor of inertia 

I = Y m 'P 2 i 1 “ m ‘PiPj ( 14 ' 48 ) 

i 

or component-wise 

Im,n — ^ ' W^iPj &m,n 772 iPi,mPi,n (14.49) 

i 

as 

L;, lt = /w. (14.50) 


14.6 Equations of Motion of a Rigid Body 


Let F, be an external force acting on m,-. Then the equation of motion for the center 
of mass is 


— 2 Y m i r i =MR = ^F, = F,„. 


(14.51) 


If there is no total external force F ext , the center of mass moves with constant velocity 


R = R„ + V(t —f 0 ). (14.52) 

The time derivative of the angular momentum equals the total external torque 

^;L = ~ Y m ‘ r i x T = Y m ‘ T ; x r i = Y r ' : x F < = Y N/ = N “' (14-53) 

i i i i 

which can be decomposed into 

= R x F exl + Y Pi x F <- (14-54) 

i 

With the decomposition of the angular momentum 
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d d d 

. I- — L COM + ~T^mt 

At At dt 

we have two separate equations for the two contributions: 

■pLcoM = —MR x R = MR x R = R x F ext 
At d? 

d x 

, T.'a;? — ^ t Pi X I' ; — N eX f R X I'i i; — N jnt 


(14.55) 


(14.56) 

(14.57) 


14.7 Moments of Inertia 


The angular momentum (14.50) is 
L, RoI — Iu> — AA 1 IAA 1 us — AlfrUib 


(14.58) 


where the tensor of inertia in the body fixed system is 
h = A~ l IA = A~' (Y.mrfpi - miPipJ^A 
= m ‘A r p' PiA - m,A r p-,p]A 

i 

= 22 m ‘Pa> ~ m >PibPl- (14.59) 

i 

Since //, does not depend on time (by definition of the body fixed system) we will 
use the principal axes of h as the axes of the body fixed system. Then //, takes the 
simple form 


Ih 0 0\ 

h = 0 h o (14.60) 

V o o h ) 

with the principle moments of inertia /| 2 ,3 - 


14.8 Equations of Motion for a Rotor 


The following equations describe pure rotation of a rigid body: 


d 

—A = WA — AW h 
At 


(14.61) 
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, — N ; „, (14.62) 

At 

( 0 -ut 3 uii \ 

W 3 0 -Wi W/y = ~£ijk ljJ k (14.63) 

—W2 Wl 0 / 

Lint = = Iul = AlfjUlf, (14.64) 

//f 1 0 0 \ 

= 4 1 ^ =1 0 I 2 1 0 1 L/,,,* w = Auj h (14.65) 

V 0 o w 

4 = const. (14.66) 


14.9 Explicit Methods 


Equation (14.61) for the rotation matrix and (14.62) for the angular momentum have 
to be solved by a suitable algorithm. The simplest integrator is the explicit Euler 
method (Fig. 14.3) [169]: 



Fig. 14.3 (Global error of the explicit methods) The equations of a free rotor (14.8) are solved 
using the explicit first order (full curves) and second order (dashed curves ) method. The deviations 
| det(A) — 1| ( diamonds ) and E^ n — £*,,,(0)1 ( circles) at t= 10 are shown as a function of the time 
step At. Full circles show the energy deviation of the first order method with reorthogonalization. 
The principal moments of inertia are If, = diag(l, 2, 3) and the initial angular momentum is 
L = (1, 1. 1). See also Problem 14.1 
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A(t + At) = A(t) + A(t)W h (t)At + 0(At 2 ) (14.67) 

L intit + At) — L int(t) + N i nl (t)At + 0(At~). (14.68) 

Expanding the Taylor series of A(t) to second order we have the second order approx¬ 
imation (Fig. 14.3) 


A(t+At) = A(t) + A(t)W h (t)At + ~ (A(t)W?;(t)+A{t)W b (t)) At 2 + 0(At 3 ). 

(14.69) 

A corresponding second order expression for the angular momentum involves the 
time derivative of the forces and is usually not practicable. 

The time derivative of W can be expressed via the time derivative of the angular 
velocity which can be calculated as follows: 

^ = 4’ 1 ( 14 - 1 ) L int + I^A- 1 N int = 

= 4“‘ {-A-'W) L,„ f +/- 1 A' 1 N,„ f = -!»' W h U nt ,b + V'N,„ a . (14.70) 

Alternatively, in the laboratory system 

ju = ^(Aw t ) = WAw b - Af-'A-' WL mt + Af-'A- l N ml 
= h/-'h(N w , - WL illt ) (14.71) 

where the first summand vanishes due to 


WAut b = AWh^b — Auj h x u) b = 0. 


(14.72) 


Substituting the angular momentum we have 


LI _ 1 

Tt^b ~ ^ ^int,b 


/-' W h I b u b 


which reads in components: 


( u b i 
Ub2 
^b3 


) 


4">*2 

W^bJ 



( 0 —UJbi UJb2 
U> b 3 0 — UJ b \ 

—UJ b 2 LObi 0 


Ib2U b 2 

IblUbl 


(14.73) 


(14.74) 
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Evaluation of the product gives a set of equations which are well known as Euler’s 
equations: 



(14.75) 


14.10 Loss of Orthogonality 

The simple methods above do not conserve the orthogonality of A. This is an effect 
of higher order but the error can accumulate quickly. Consider the determinant of A. 
For the simple explicit Euler scheme we have 

det(A + A A) = det(A + WAAt ) = detA det(l + W At) — detA (1 + ur At 2 ). 

(14.76) 

The error is of order At 2 , but the determinant will continuously increase, i.e. the 
rigid body will explode. For the second order integrator we find 

det(A + A A) = det l A + WAAt + — (W 2 A + WA) J 



(14.77) 


This can be simplified to give 


det(A + AA) = detA (1 + ututAt 3 + •••)■ 


(14.78) 


The second order method behaves somewhat better since the product of angular 
velocity and acceleration can change in time. To assure that A remains a rotation 
matrix we must introduce constraints or reorthogonalize A at least after some steps 
(for instance every time when | det (A) — 11 gets larger than a certain threshold). The 
following method with a symmetric correction matrix is a very useful alternative 
[170]. The non-singular square matrix A can be decomposed into the product of an 
orthonormal matrix A and a positive semi-definite matrix S 


A = AS 


(14.79) 


with the positive definite square root of the symmetric matrix A 7 A 


S = (. A r A) 1/2 


(14.80) 
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(14.81) 


and 

A = AS -1 = A(A T A)- 1/1 
which is orthonormal as can be seen from 

A t A = (S~ 1 ) t A t AS~ l = S~ l S 2 S~ l = 1. (14.82) 

Since the deviation of A from orthogonality is small, we make the approximations 
S=l + s (14.83) 


A T A = S 2 ^ 1 + 2s 


A t A — 1 


s~ l K 1 -j PS 1 


1 -A 1 A 


(14.84) 

(14.85) 

(14.86) 


which can be easily evaluated. 


14.11 Implicit Method 


The quality of the method can be significantly improved by taking the time derivative 
at midstep (Fig. 14.4) (13.5): 


Fig. 14.4 (Global error of 
the implicit method) The 
equations of a free rotor 
(14.8) are solved using the 
implicit method. The 
deviations | det(A) — 11 
(diamonds) and 
I E kin - E kin ( 0)1 ( circles) at 
t = 10 are shown as a 
function of the time step At. 
Initial conditions as in 
Fig. 14.3. See also 
Problem 14.1 
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A(t + At) = A(t) + A 

^-‘intA At) — L/ nf (r) + N int 
Taylor series expansion gives 


(■*?) 


At - 


At + • • ■ . 


4 + fM ,+ f) 


— I At 


At - 


At 2 


= A(t)W(t)At+A(t)W(t)— + A(t)W(j) — + 0(At ) 
= A{t)W(t)At + (A(t)W\t) + A(t)W{t))— + 0(At 3 ) 


(14.87) 

(14.88) 


(14.89) 

(14.90) 


which has the same error order as the explicit second order method. The matrix 
A(t + ^f ) at mid-time can be approximated by 


~(A(t) + A(t + At)) 


/ At\ At 2 -/ At\ / At\ 

= A ( f+ T) + ^ A ( ?+ T) + -= A ( ?+ T) 


— \+0(At 2 ) (14.91) 


which does not change the error order of the implicit integrator which now becomes 


1 


K) 


A(t + At) = A(t) + - ( A{t ) + A(t + At)) W \ t + — ) At + 0{At A ). (14.92) 

This equation can be formally solved by 

M, + An = m .)( i + t" , ( , + t))( i "I w (' + t)) ' = (f)■ 

(14.93) 

Alternatively, using angular velocities in the laboratory system we have the similar 
expression 


A(t + At) = 


i_ Sri/- . 


/ 1 f At / / At\ 

V + t)J [ l + T w { t+ Y)r ) = T {Y) 


A(t) = T\ — ]A(t). 

(14.94) 


The angular velocities at midtime can be calculated with sufficient accuracy from 


W 


M) 


At . 


— ) = W(t) + —W(t) + 0(At~). 


(14.95) 
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With the help of an algebra program we easily prove that 


det 


(l + fw)=de,(l-fw) 


1 + 


u> 2 At 2 

4 


(14.96) 


and therefore the determinant of the rotation matrix is conserved. The necessary 
matrix inversion can be easily done: 



f + unut 2 ^ 

f + WiW3^j- 


At | ... At 1 

T + ^2 — 
,2 /A ,2 


-W3 T 

1 + 

wt x + w 2 w 3 4f 


\ 

—LO\ ^ + UJ2UJ-}^- 

1 + ^ J 


l+ut 2 ^' 


(14.97) 


The matrix product is explicitly 


z\r 

Zif 

[ 1 + tH 

1- 

1 

to | 

1_ 


1 H———— —At 2 — uJbiAt + ujb\Ub2^=r ^JbiAt + u>biu>b3^r- ^ 

^b^At + LL>b\UJb2^r 1 H- M+ 4 m —— 4lr 2 —ujbiAt + ajbi^bi^r- 

—Ldb2At + U>blU>b3^T~ UblAt + 1H- M 4 m+ m At~ ) 


X 


1 


i + ^ 2 t 


(14.98) 


With the help of an algebra program it can be proved that this matrix is even orthog¬ 
onal 


TlT h = 1 (14.99) 

and hence the orthonormality of A is conserved. The approximation for the angular 
momentum 


L int(t) + N,-„, H ——J At 

. At 2 

— L/n/(t) + N/„/(t)Zlf + N) nf (f) ——!-■■■ = L ,-,„(? + At) + 0(At 3 ) (14.100) 
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can be used in an implicit way 


4“ At) T Njw(i) i 

L in,(t + At) = UAt) + —- J - —At + 0(At 3 ). (14.101) 

Alternatively Euler’s equations can be used in the form [171, 172] 

( /\t\ ( At\ I b2 —7 W TV^i 

t + —J = UJbl I t - — \ + — 1 - U} b 2(t)uJb3(t)At + — zlf etc - 

(14.102) 


where the product is approximated by 


1 

Ubi(t)uJb3(t) = - 


Ub2 


( At \ { 

At) 

/ At\ ( zlAl 


'"tJ 



(14.103) 


ut b \ (t + A ) is determined by iterative solution of the last two equations. Starting 
with — 40 convergence is achieved after few iterations. 


14.12 Example: Free Symmetric Rotor 


For the special case of a free symmetric rotor (I b 2 = hi , N;,„ = 0) Euler’s equations 
simplify to: 

u>bi = 0 (14.104) 


t0>3 


42(3) — 4l 
42(3) 

I hi ~ 42(3) 


^>b\^>bi — AuJ bi 
UblUbl = — Aw*2 


iblO) 


, 42(3) — 4l 

A = --- UJbl- 

42(3) 


(14.105) 

(14.106) 

(14.107) 


Coupled equations of this type appear often in physics. The solution can be found 
using a complex quantity 


C2 = u) b 2 + ibJbi 


(14.108) 


which obeys the simple differential equation 
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£2 = u>bi + iu>b3 — ~i(i^u>bi + A wh) = —i\£2 


(14.109) 


with the solution 

fi = ,Qoe“ ! ' A '- (14.110) 

Finally 


/ w*i(0) \ / Wfei(0) \ 

u>* = I 91(^2 0 e _ ' A ') I = I w i2 (0) cos(At) + w M (0) sin(Af) I (14.111) 

\^(S2 0 e~ iXt ) J \wm( 0) cos(Ar) - w 42 (0) sin(Af)/ 

i.e. ujb rotates around the 1-axis with frequency A. 


14.13 Kinetic Energy of a Rotor 

The kinetic energy of the rotor is 

yT 2 = Xy (R + ' 4 ^) 

i i 

Z ITli • rp T ' T ' ' 'V'-' / 71 ; rp • rp • 

y (* r + prX)0R + Ap a ) = yP 2 + X -^PaA Apib- (14.112) 

i i 

The second part is the contribution of the rotational motion. It can be written as 

Z /?2 i rp T T ^ 1 /?7 IT O 1 7 1 

^plW T b A r AW h p ib = - X ypfXP* = (14.113) 

Z / 

since 

- W WWJ2 \ 

—UJbliObl —UblUbl I = ^b — b<^>b ■ (14.114) 

-WmWw -LO b 2U>b3 Wj! + W* 2 / 


14.14 Parametrization by Euler Angles 

So far we had to solve equations for all 9 components of the rotation matrix. But there 
are six constraints since the column vectors of the matrix have to be orthonormalized. 
Therefore the matrix can be parametrized with less than 9 variables. In fact it is 
sufficient to use only three variables. This can be achieved by splitting the full rotation 
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into three rotations around different axis. Most common are Euler angles defined by 
the orthogonal matrix [173] 


( cos %(’ cos <p — cos 9 sin </> sin ip 
cos ip sin <p + cos 9 cos <p sin ip 
sin 9 sin ip 


sin ip cos <p — cos 9 sin cp cos ip sin 9 sin <p \ 
sin ip sin cp + cos 9 cos cp cos ip — sin 9 cos cp | 
sin 9 cos ip cos 9 J 

(14.115) 


obeying the equations 
sin cp cos 9 


cos cp cos 9 
UJ V -:—“-b 


sin 9 y sin 9 

9 = t o x cos <p + u) y sin cp 

sin <p cos cp 
ip = lo x — 


(14.116) 

(14.117) 

(14.118) 


sin 9 ' sin 9 

Different versions of Euler angles can be found in the literature, together with 
the closely related cardanic angles. For all of them a sin 9 appears in the denomi¬ 
nator which causes numerical instabilities at the poles. One possible solution to this 
problem is to switch between two different coordinate systems. 


14.15 Cayley-Klein-Parameters, Quaternions, 

Euler Parameters 

There exists another parametrization of the rotation matrix which is very suitable for 
numerical calculations. It is connected with the algebra of the so called quaternions. 
The vector space of the complex 2x2 matrices can be spanned using Pauli matrices 
by 


1 = 


(i?) -P PP PP 


(14.119) 


Any complex 2x2 matrix can be written as a linear combination 

col+ccr. (14.120) 

Accordingly any vector x e B? can be mapped onto a complex 2x2 matrix: 

x - P=( P *- f ?y (14.121) 

\x + ly -z J 

Rotation of the coordinate system leads to the transformation 


P' = QPQ' 


(14.122) 
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where 

Q = 


(?!) 


(14.123) 


is a complex 2x2 rotation matrix. Invariance of the length (|x| = y/— det(P)) under 
rotation implies that Q must be unitary, i.e. Q T = Q~ x and its determinant must be 
1. Explicitly 


( a '' r ) = Q-‘ = _ 1 _( 4 ~f>\ 

\(3* 6* J aS — /?7 \— 7 a ) 


G f = 

and Q has the form 


Q = 


( a (3\ 
\-/3* a*) 


with \a\~ + 1/31 = 1. 


(14.124) 


(14.125) 


Setting x± = x ± iy, the transformed matrix has the same form as P: 


QPQ 


\/3\ 2 )z —0 1 x + + a 2 x _ — 2a/3z 

a* 2 x + — /3* 2 x- — 2a*(3*z —a*/3x + — a/3*x- — {\a\ 2 — \/3\ 2 )z. 


_ / a*/3x + + f3*ax _ + (|a 
— ^ a* 2 x + 

-us> 


(14.126) 


From comparison we find the transformed vector components: 

J = \(x+ + x'_) = l -(a* 2 - i3 2 )x+ + ^(a 2 - p* 2 )x- - (a/3 + a*/3*)z 

a* 2 + a 2 — (3*2 — /3 2 i(a * 2 - a 2 + f3* 2 - p 2 ) , ^ 

=---jc 4--- y - (a/3 + a p )z (14.127) 

/ = — x(_) = —(a* 2 + /3 2 )jc_|_ + : , ! .(-/3* 2 - a 2 ).r_ + -/ (—a*/3* + a/3)z 

Q * 2 — a 2 — /3* 2 + /3 2 a* 2 + a 2 + /3* 2 + /3 2 , ^ m 

= -* 4-y 4- i(a (3 — ap)z (14.128) 

2i 2 

z / = (a*/3 4- ap*)x + i(a*( 3 - a/3*)y 4- (|a| 2 - |/3| 2 )z. (14.129) 

This gives us the rotation matrix in terms of the Cayley-Klein parameters a and /3: 


A = 


I a * 1 +°‘ 1 -P* 2 -l 32 i(a* 2 -a 2 +p* 2 -/3 2 ) _ (a/3 + a *p* ) \ 

a t2 -a 2 -P* 2 +0 2 a ,2 +a 2 +p ,2 +l3 2 1 

y (a*/3 + a/3*) i(a*/3 - a/3*) (|a| 2 - |/3| 2 ) y 


}(—a*/3* + a/3) 


(14.130) 
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For practical calculations one often prefers to have four real parameters instead of 
two complex ones. The so called Euler parameters t/o, qu q 2 , <73 are defined by 

a = q 0 + iq 3 (3 = q 2 + iq\. (14.131) 


Now the matrix Q 

r. ( qo + iqi qi + iq\\ ... . . . • 

Q= I „ , ■ „ • I = q 0 l + iqicr x + iq 2 a y + iq 3 a z (14.132) 

becomes a so-called quaternion which is a linear combination of the four matrices 


U = 1 I = ia z J = ia y K — ia x (14.133) 

which obey the following multiplication rules: 

I 2 =J 2 = K 2 = -U 
IJ = —JI = K 
JK = -KJ = I 

KI = —IK = J. (14.134) 

In terms of Euler parameters the rotation matrix reads 

( ql + q\-q\-ql 2(q\q 2 + qm) ^ 2.(qiq 3 - q 0 q 2 ) \ 

2(<7i<72 - 9093 ) ql - q 2 i + ql - 2(4 2 <?3+ <7o<7i) ) (14.135) 

2(q l q 2 + q 0 q 2 ) 2(q 2 q 2 - q 0 q x ) q 2 0 - q\ - q\ + q\) 


and from the equation A — WA we derive the equation of motion for the quaternion 


/ qo \ 


/ 0 U>\ lo 2 \ 


q o\ 

qi 

l 

—U)\ 0 — UJ 2 U! 2 


q i 

qi 

~ 2 

— UJ 2 Ul 2 0 —bJ\ 


qi 

\qi) 


y — lot, — L0 2 Wi 0 y 


W) 


(14.136) 


or from A — AW), the alternative equation 


qo\ 


( 0 ui\b uj 2 b u> 2 b \ 


qo \ 

qi 

l 

—UJlb 0 UJ 2 b —U 2 b 


q\ 

qi 

~~ 2 

—UJ 2 b —W3 b 0 LO\b 


qi 

VW 


y—LVlb U 2 b —LO\b 0 ) 


WJ 


(14.137) 


Both of these equations can be written briefly in the form 


q = W q. 


(14.138) 
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Example: Rotation Around the z-axis 

Rotation around the z-axis corresponds to the quaternion with Euler parameters 


/ cosf \ 
0 
0 

V-sinf / 


(14.139) 


as can be seen from the rotation matrix 


'(cosf) 2 -(sinf) 2 
A = \ 2 cos y sin y 

0 

' cos cot — sin wr O' 
sinojf cos Lot 0 

0 0 1 


, , ° \ 

(cos f ) 2 - (sin f ) 2 ^ 0 

0 (cosf) 2 + (sinf) 2 / 


(14.140) 


The time derivative of q obeys the equation 



3 

o 

o 

o 


/ cos f \ 


/-fsinwA 

1 

0 0 — co 0 


0 


0 

2 

0 CO 0 0 


0 


0 


y —co 0 0 0 j 


{-sinf) 


y — y COS cot j 


(14.141) 


After a rotation by 2 tt the quaternion changes its sign, i.e. q and —q parametrize the 
same rotation matrix! 


14.16 Solving the Equations of Motion with Quaternions 


As with the matrix method we can obtain a simple first or second order algorithm 
from the Taylor series expansion 


q (t + At) = q(r) + W(t)q(t)At + (W(t) + W 2 (t))<\{t)— + ■ ■ • . (14.142) 

Now only one constraint remains, which is the conservation of the norm of the 
quaternion. This can be taken into account by rescaling the quaternion whenever its 
norm deviates too much from unity. 

It is also possible to use Omelyan’s [174] method: 

q (t + At) = q(f) + W {t + — ^ -(q(0 + q(? + At)) 


(14.143) 
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gives 


q(t+At)= ( 1 + Y W ) 


where the inverse matrix is 


At 

1 + —W)q(r) 


/ At 1 / At ~\ 

V 2 ) l+ w 2 ^l 2 ) 


and the matrix product 


/ At ~\ 1 / At ~\ 1 -cu 2 4£ 

(1-W) (l + — w)= - ^ 

V 2 / V 2 ) 1+u; 2 # 


+ 


At 


1 + ui- 


2 At 2 


W. 


(14.144) 


(14.145) 


(14.146) 


16 " 1 " 16 

This method conserves the norm of the quaternion and works quite well. 


Problems 

Problem 14.1 Free Rotor 

In this computer experiment we compare different methods for a free rotor (Sect. 14.8, 
Fig. 14.5): 

• explicit first order method (14.67) 

A{t+ At) = A(t) + A(t)W b (t)At + 0(At 2 ) (14.147) 

• explicit second order method (14.69) 


Fig. 14.5 Free asymmetric 
rotor 
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Fig. 14.6 Rotor in an i i A A A A 

electric field !!!!!! 



A(t + At) = A(t) + A(t)W b (t)At + ~ (A(t)W 2 (t) +A(t)W b (t )) At 2 + 0(At 3 ) 

(14.148) 

• implicit second order method (14.93) 

A(t+At) = A(o (i + y w ( f + y)) (i - y w (* + y)) + 0(Af3 y 

(14.149) 

The explicit methods can be combined with reorthogonalization according to 
(14.79) or with the Gram-Schmidt method. Reorthogonalization threshold and time 
step can be varied and the error of kinetic energy and determinant are plotted as a 
function of the total simulation time. 

Problem 14.2 Rotor in a Field 

In this computer experiment we simulate a molecule with a permanent dipole moment 
in a homogeneous electric field E (Fig. 14.6). We neglect vibrations and describe the 
molecule as a rigid body consisting of nuclei with masses /«, and partial charges Q,. 
The total charge is Y, Qi = 0- The dipole moment is 

P = X G ' r , (14.150) 

i 

and external force and torque are 


(14.151) 


F ext — 'y \ 2/E — 0 

i 

N «, = X 2' r ' x E = p x E. 


(14.152) 
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Fig. 14.7 Molecular 
collision 



o 


The angular momentum changes according to 


d 

At 


L/'/ic — p x E 


(14.153) 


where the dipole moment is constant in the body fixed system. We use the implicit 
integrator for the rotation matrix (14.93) and the equation 


w&(0 = ~I b 1 Wb(t)L in ,'b(t) + 4 l A 1 (f)(p(f) x E) 


(14.154) 


to solve the equations of motion numerically. 

Obviously the component of the angular momentum parallel to the field is con¬ 
stant. The potential energy is 



(14.155) 


Problem 14.3 Molecular Collision 

This computer experiment simulates the collision of two rigid methane molecules 
(Fig. 14.7). The equations of motion are solved with the implicit quaternion method 
(14.143) and the velocity Verlet method (13.11.4). The two molecules interact by a 
standard 6-12 Lennard-Jones potential (15.24) [163]. For comparison the attractive 
r 6 part can be switched off. The initial angular momenta as well as the initial 
velocity v and collision parameter b can be varied. Total energy and momentum are 
monitored and the decomposition of the total energy into translational, rotational and 
potential energy are plotted as a function of time. 

Study the exchange of momentum and angular momentum and the transfer of 
energy between translational and rotational degrees of freedom. 




Chapter 15 

Molecular Mechanics 


Classical molecular mechanics simulations have become a very valuable tool for 
the investigation of atomic and molecular systems [175-179], mainly in the area of 
materials science and molecular biophysics. Based on the Bom—Oppenheimer sep¬ 
aration which assumes that the electrons move much faster than the nuclei, nuclear 
motion is described quantum mechanically by the Hamiltonian 

H = [T Nuc + U (rf“ c )]. (15.1) 

Molecular mechanics uses the corresponding classical energy function 

( p Nuc ) 2 

T N “C + v (j .Nucj = ^ n - + JJ ( r Nucj ( 15 .2) 

j 1 


which treats the atoms as mass points interacting by classical forces 

F, = -grad r . U (r"“ e ). (15.3) 


Stable structures, i.e. local minima of the potential energy can be found by the 
methods discussed in Chap. 6. Small amplitude motions around an equilibrium geom¬ 
etry are described by a harmonic normal mode analysis. Molecular dynamics (MD) 
simulations solve the classical equations of motion 


d 2 r i 

»!,— = F t = -grad t .U 


(15.4) 


numerically. 

The potential energy function U can be calculated with simplified quan¬ 

tum methods for not too large systems [180, 181]. Classical MD simulations for 
larger molecules use empirical force fields, which approximate the potential energy 
surface of the electronic ground state. They are able to describe structural and con¬ 
formational changes but not chemical reactions which usually involve more than one 
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Fig. 15.1 (Molecular 
coordinates) Cartesian 
coordinates (Left) are used 
to solve the equations of 
motion whereas the potential 
energy is more conveniently 
formulated in internal 
coordinates (Right) 




electronic state. Among the most popular classical force fields are AMBER [182], 
CHARMM [183] and GROMOS [184, 185], 

In this chapter we discuss the most important interaction terms, which are con¬ 
veniently expressed in internal coordinates, i.e. bond lengths, bond angles and di¬ 
hedral angles. We derive expressions for the gradients of the force field with respect 
to Cartesian coordinates. In a computer experiment we simulate a glycine dipeptide 
and demonstrate the principles of energy minimization, normal mode analysis and 
dynamics simulation. 


15.1 Atomic Coordinates 

The most natural coordinates for the simulation of molecules are the Cartesian co¬ 
ordinates (Fig. 15.1) of the atoms, 

r, = (xj, y it Zi) (15.5) 

which can be collected into a 3/V-dimensional vector 


(6-6- --&n) = (xi, yi, zi, x 2 ■ ■ ■ x N , y N , zn)- (15.6) 

The second derivatives of the Cartesian coordinates appear directly in the equations 
of motion (15.4) 

m,fi r = F r r=l---31V. (15.7) 

Cartesian coordinates have no direct relation to the structural properties of molecules. 
For instance a protein is a long chain of atoms (the so called backbone) with additional 
side groups (Fig. 15.2). 

The protein structure can be described more intuitively with the help of atomic dis¬ 
tances and angles. Internal coordinates are (Fig. 15.3) distances between two bonded 
atoms (bond lengths) 
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Fig. 15.2 (Conformation of 
a protein) The relative 
orientation of two successive 
protein residues can be 
described by three angles 
(<F, 0, u) 



Fig. 15.3 (Internal 
coordinates) The structure of 
a molecule can be described 
by bond lengths, bond angles 
and dihedral angles 



b ij = \T ij \ = \r i -Tj\, (15.8) 

angles between two bonds (bond angles) 



and dihedral angles which describe the planarity and torsions of the molecule. A 
dihedral angle (Fig. 15.4) is the angle between two planes which are defined by three 
bonds 


Oijki = sign {Qijki) arccos(n i7i n ;W ) 


(15.10) 
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Fig. 15.4 Dihedral angle 


Fig. 15.5 (Glycine 
dipeptide model) The 
glycine dipeptide is the 
simplest model for a peptide. 
It is simulated in 
Problem 15.1. Optimized 
internal coordinates are 
shown in Table 15.1 


0 



IJ13 

H10 —Nl—C2- H12 


/ 

Hll 


C3 ■ 


07 


H14 

/ 

-N4 

\ 

H15 H16 


08 


C6 


09 —H17 


r ij x r kj r kj x r H 

—-1 n jki = |- 

r i; x Tkj\ \r kj xr k i 

where the conventional sign of the dihedral angle [186] is determined by 

signet/ = sign (r kj (n ijk x n ja )) . (15.12) 

Internal coordinates are very convenient for the formulation of a force field. On the 
other hand, the kinetic energy (15.2) becomes complicated if expressed in internal 
coordinates. Therefore both kinds of coordinates are used in molecular dynamics 
calculations. The internal coordinates are usually arranged in Z-matrix form. Each 
line corresponds to one atom i and shows its position relative to three atoms j, k. I in 
terms of the bond length bjj, the bond angle (j)ij k and the dihedral angle Oijki (Fig. 15.5 
and Table 15.1). 
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Table 15.1 (Z-matrix) The optimized values of the internal coordinates from Problem 15.1 are 
shown in Z-matrix form. Except for the first three atoms the position of atom i is given by its 
distance bij to atom j, the bond angle <f>ij k and the dihedral angle djjki 


Number i 

Label 

j 

k 

1 

Bond 
length 
bij (A) 

Bond 
angle <j> ijk 

Dihedral 

dijkl 

1 

N1 







2 

C2 

1 



1.45 



3 

C3 

2 

1 


1.53 

108.6 


4 

N4 

3 

2 

1 

1.35 

115.0 

160.7 

5 

C5 

4 

3 

2 

1.44 

122.3 

-152.3 

6 

C6 

5 

4 

3 

1.51 

108.7 

-153.1 

7 

07 

3 

2 

1 

1.23 

121.4 

-26.3 

8 

08 

6 

5 

4 

1.21 

124.4 

123.7 

9 

09 

6 

5 

4 

7.34 

111.5 

-56.5 

10 

H10 

1 

2 

3 

7.02 

108.7 

-67.6 

11 

HI 1 

1 

2 

3 

1.02 

108.7 

49.3 

12 

H12 

2 

3 

4 

1.10 

109.4 

-76.8 

13 

H13 

2 

3 

4 

1.10 

109.4 

38.3 

14 

H14 

4 

3 

2 

1.02 

123.1 

27.5 

15 

H15 

5 

4 

3 

1.10 

111.2 

-32.5 

16 

H16 

5 

4 

3 

1.10 

111.1 

86.3 

17 

H17 

9 

6 

5 

0.97 

106.9 

-147.4 


15.2 Force Fields 

Classical force fields are usually constructed as an additive combination of many 
interaction terms. Generally these can be divided into intramolecular contributions 
Ubonded which determine the configuration and motion of a single molecule and 
intermolecular contributions U non -b on ded describing interactions between different 
atoms or molecules 


U — Ubonded non—bonded- 


(15.13) 


15.2.1 Intramolecular Forces 

The most important intramolecular forces depend on the deviation of bond lengths, 
bond angles and dihedral angles from their equilibrium values. For simplicity a sum 
of independent terms is used as for the CHARMM force field [183, 187, 188] 
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(15.14) 


The forces are derived from potential functions which are in the simplest case ap¬ 
proximated by harmonic oscillator parabolas (Fig. 15.6), like the bond stretching 
energy 

u u nd = l k vtoj- b ij> 2 ( 15 - 15 ) 

angle bending terms 

U™* le = l -k ijk (<t>ij k -<t>° jk ) 2 (15-16) 

together with the Urey-Bradly correction 

= \k ijk {b ik - b« k ? 


(15.17) 
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and “improper dihedral” terms which are used to keep planarity 

Torsional energy contributions are often described by a cosine function 1 

= kiju (1 - cos {mO m - 9° ijkl j) (15.19) 

where m = 1, 2, 3, 4, 6 describes the symmetry. For instance m — 3 for the three 
equivalent hydrogen atoms of a methyl group. In most cases the phase shift = 0 


or 0 ( -j kl — 7 r. Then the dihedral potential can be expanded as 
for instance 

a polynomial of cos 9, 

m=l: U?;k dra ' = k{ licosflyw) 

(15.20) 

m=2: Uffi dral = k± HI - 2 (cos 6 ijkl ) 2 ) 

(15.21) 

m=3: U d jtf dral = *(1 ± 3 cos 9 ijkl T 4(cos 9 ljkl ) 3 ). 

(15.22) 

For more general 9^ kl the torsional potential can be written as a polynomial of 


cos e ijk i and sin () J;W . 

The atoms are classified by element and bonding environment. Atoms of the same 
atom type are considered equivalent and the parameters transferable (for an example 
see Tables 15.2, 15.3, 15.4). 


15.2.2 Intermolecular Interactions 

Interactions between non-bonded atoms 

Unon-bonded = U C ° U ' + U vdW (15.23) 

include the Coulomb interaction and the weak attractive van der Waals forces which 
are usually combined with a repulsive force at short distances to account for the Pauli 
principle. Very often a sum of pairwise Lennard-Jones potentials is used (Fig. 15.7) 
[163] 


1 Some force-fields like Desmond [189] or UFF [190] use a more general sum k Xm=o c '« cos ( m $ — 

9°). 
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U 


vdw 


z z <• 

A^B ieAJeB 


A^B ij 



(15.24) 


The charge distribution of a molecular system can be described by a set of multi¬ 
poles at the position of the nuclei, the bond centers and further positions (lone pairs 
for example). Such distributed multipoles can be calculated quantum chemically for 
not too large molecules. In the simplest models only partial charges are taken into 
account giving the Coulomb energy as a sum of atom-atom interactions 


jjCoul 


4n£ 0 r,: 

AjtBieAJeB u 


(15.25) 


More sophisticated force fields include higher charge multipoles and polarization 
effects. 


15.3 Gradients 

The equations of motion are usually solved in Cartesian coordinates and the gradients 
of the potential are needed in Cartesian coordinates. Since the potential depends only 
on relative position vectors rthe gradient with respect to a certain atom position 
r/ f can be calculated from 


Table 15.2 (Atom types of the glycine dipeptide) Atom types for glycine oligopeptides according 
to Bautista and Seminario [191]. The atoms are classified by element and bonding environment. 
Atoms of the same atom type are considered equivalent 


Atom type 

Atoms 

C 

C3 

Cl 

C2, C5 

c 2 

C6 

N 

N4 

n 2 

N1 

0 

07 

Oi 

09 

0 2 

08 

H 

H14 

Hi 

H12, H13, H15, H16 

h 2 

H17 

Hi 

H10.H11 
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Table 15.3 (Bond stretching parameters) Equilibrium bond lengths (A) and force constants 
(kcal moL'A" 2 ) for the glycine dipeptide from [191] 


Bond type 

b° 

k 

Bonds 

r C,N 

1.346 

1296.3 

C3-N4 

rci,N 

1.438 

935.5 

N4-C5 

rci,N2 

1.452 

887.7 

N1-C2 

rci,ci 

1.510 

818.9 

C5-C6 

r C,C\ 

1.528 

767.9 

C2-C3 

rc 2,02 

1.211 

2154.5 

C 6 -O 8 

r c,o 

1.229 

1945.7 

C3-07 

r C2,0\ 

1.339 

1162.1 

C6-09 

tn,h 

1.016 

1132.4 

N4-H14 

rN2,H3 

1.020 

1104.5 

N1-H10, Nl-Hl 1 

rci.Hl 

1.098 

900.0 

C2-H12, C2-H13, 
C5-H15,C5-H16 

roi.H2 

0.974 

1214.6 

09-H17 


Fig. 15.7 (Lennard-Jones 
potential) The 6-12 potential 
(15.24) has its minimum at 
r m in = -^cr ~ 1.1 2a with 

6'iriiri = e 



g rad r fc = ^(S ik - S jk ) grad r ... 
i<j 


(15.26) 


Therefore it is sufficient to calculate gradients with respect to the difference vectors. 
Numerically efficient methods to calculate first and second derivatives of many force 
field terms are given in [192-194]. The simplest potential terms depend only on 
the distance of two atoms. For instance bond stretching terms, Lennard-Jones and 
Coulomb energies have the form 


Uij = U(nj ) = U( |r, 7 |) 


(15.27) 
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Table 15.4 (Bond angle parameters) Equilibrium bond angles (deg) and force constants 
(kcal mol^'rad -2 ) for the glycine dipeptide from [191] 


Angle type 

4,° 

k 

Angles 

<t>N,C,Cl 

115.0 

160.0 

C2-C3-N4 

<Pci,N,C 

122.3 

160.1 

C3-N4-C5 

<t>C\,C2.0\ 

111.5 

156.0 

C5-C6-09 

<l>Cl,C2,02 

124.4 

123.8 

C5-C6-08 

<t>Cl,C,0 

121.4 

127.5 

C2-C3-07 

<p02.C2,0l 

124.1 

146.5 

08-C6-09 

<t>N,C,0 

123.2 

132.7 

N4-C3-07 

<Pc,Cl.Hl 

110.1 

74.6 

H12-C2-C3, 

H13-C2-C3 

<t>C2,Cl,Hl 

109.4 

69.6 

H16-C5-C6, 

H15-C5-C6 

<t>C,N,H 

123.1 

72.0 

C3-N4-H14 

<Pci.N,H 

114.6 

68.3 

C5-N4-H14 

<Pci.N2.H3 

108.7 

71.7 

H10-N1-C2, 

HI 1-N1-C2 

4>H1,C1,H1 

106.6 

48.3 

H13-C2-H12,H15-C5- 

H16 

<pH3,N2,H3 

107.7 

45.2 

H10-N1-H11 

<t>C,C\,N2 

109.0 

139.8 

N1-C2-C3 

<Pc 2 ,Cl,N 

108.6 

129.0 

N4-C5-C6 

<l>C2,Ol.H2 

106.9 

72.0 

H17-09-C6 

<pN,Cl,Hl 

111.1 

73.3 

H15-C5-N4, 

H16-C5-N4 

<pN2,Cl,Hl 

112.6 

80.1 

H13-C2-N1, 

H12-C2-N1 


where the gradient is 


gradr ,jUij = 


dU r U 

dr |r, 7 f 


The most important gradients of this kind are 



(15.28) 


(15.29) 


grad r ,^f = 24s u (-2^ + ^-)r i j 
\ ij ij / 


(15.30) 
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grad r Ufj° ul = - 


mj 

47 T £ 0 r. 


3 r, 7 • 


(15.31) 


The gradient of the harmonic bond angle potential is 

gradrt/""^ = k ( ( Pijk ~ (p°)g™d r <fi ijk (15.32) 

where the gradient of the angle can be calculated from the gradient of its cosine 


grad r 4>ij k = -- 


1 


= —L_ (- 

sm cpijk \|r 


l r A, 


grad cos <j> ijk = —— l — ( — 
1 sm <p ljk Wn 

) 


r kj 


sin (j> ijk 

r k j cos (f) ijk 


|r*/ 




r U r ki _ \ 

ruPinv 7 

(15.33) 


grad, 


t k j9ijk 


1 / r d 

sin (pijk \|r,;||r^ 


cos 4> ljk 

W r ‘' 


) 


(15.34) 


In principle, the sine function in the denominator could lead to numerical problems 
which can be avoided by treating angles close to 0 or tt separately or using a function 
of cos 4>ij k like the trigonometric potential 


U-j7' e = ^*y*(cos0y* - COS (fljk) 2 


(15.35) 


instead [190, 195, 196]. Alternatively, the gradient of <f> can be brought to a form 
which is free of singularities by expressing the sine in the denominator by a cosine 
[197] 


grad rtj </> ijk = - 





r U r kj - (r,/r^/)r„ 
r ij\]( r u r kj ~ (r,/r A 7 Tr , ; ) 2 


1 r jj x {Tkj x r jj) 

r ij |r,7 x Ca/ x r, 7 )| 


and similarly 


g rad r kj <t>ijk - ~ 


1 r k j x (r, v x r^) 
r kj \r kj x (r, 7 x r kj )\ 


(15.36) 


(15.37) 


Gradients of the dihedral potential are most easily calculated for 0 a l]kj = 0 or n. 
In that case, the dihedral potential is a polynomial of cos 9ij k i only (15.20)-(15.22) 
and 
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gradr U\j U 


dihedral 


at j dihedral 
aU ijkl 

d COS 6i jk l 


grad r cos d ijk , 


(15.38) 


whereas in the general case 0 < 9/j k i < ~ application of the chain rule gives 


grad Uf- kl edral — mkijia sin (m(9 ijk , - 9°)) grad 9 llki . (15.39) 

If this is evaluated with the help of 


grad 6j Jk i = - 


1 

sin Qua 


grad cos e iJkl 


(15.40) 


singularities appear for 9 = 0 and n. The same is the case for the gradients of the 
harmonic improper potential 


g rad U"”£ roper = k(9 ijk i - 9° ijkl ) grad 9 m . (15.41) 

Again, one possibility which has been often used, is to treat angles close to 0 or 7r 
separately [188]. However, the gradient of the angle 9jj k / can be calculated directly, 
which is much more efficient [198]. 

The gradient of the cosine follows from application of the product rule 

( rx r k j r ki x r k i \ 

(15.42) 

I r u X Tkj I I r kj X T ki \J 

First we derive the differentiation rule 

grad a [(a x b)(c x d)] = grad a [(ac)(bd) - (ad)(be)] 

= c(bd) - d(bc) = b x (c x d) (15.43) 

which helps us to find 


grad,...(r,, x r kj )(r kj x r kI ) = r kj x (r kj x r H ) 


(15.44) 


grad, t , (ry x r kj )(r kj x r kl ) = r kj x (r tj x r, 7 ) 


(15.45) 





15.3 Gradients 


363 


grad (r , 7 x r kj )(r kj x r w ) = r u x (r , 7 x r kj ) + r , 7 x (r w x r kj ). (15.46) 


and 


grad, - 


I r ij X r A 7 '| 


r kj x (r , 7 x r kj ) 

\ Y U X T kj 1 3 


(15.47) 


grad 


rtj 


r a x r i7 


r , 7 x (r fe/ x r i7 ) 


I r ij x r w ] 


(15.48) 


grad 


rtj 


*kj x r H 


r id x (n 7 x r kl ) 
\r k j x r«| 


(15.49) 


grad, 


r kj x (r w x r kj ) 


l r */ xr «l \r kj XT kl \ 3 

Finally we collect terms to obtain the gradients of the cosine [197] 


(15.50) 


rr,- x (r k i x r k i) r k : x (r,-,- x r k ; ) 
grad r .. cosfyw = - ,2 cos M 


x r y J ir kj x r H | 


r */f 


(15.51) 


= t -—-r x ( n/ W - n ij k cosd) = T ---, x ( n jki - ii, ;7 (n / 7 /n , 77 )) 


r */l 


r ^l 


r kj 


r kj\ 


x (n, 77 . x (n j ki x ny*)) 


r */ 


«•;; x r kj \ 


sin# 1 


r kj l r O xr */l 


( 1 \ sin# r^; 

ny* x —(-rjy)sm# I = - t ---r x (n iJk x (—r* 7 )) = 

r kj / r kj \ r ij x r k j | 

(-%*>■&) = 


, a r kj X (r kj X Tjj) r kj x (r k , x r kj ) 

grad cos Q m = ---ry--—,- - -cos Q m (15.52) 
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Tkj 


|r k j x r k i 


x (n j k i x (n ljk x n jk i )) 
























364 


15 Molecular Mechanics 


= -t———t x (n jk , x (~ J sin#)) 
\r k jxr k i\ \ \r kj )) 

r kj sin 9 

= —]— - r n jk i 

\ r kj x r kl\ 


sin t 


r kj \r kj x y u 


r kj x (n jk , x r kj ) 


grad rt/ cos 9 ijkl 
r a x (r kj x r,,) 


r u x (r„ x r kj ) + Tij x (r A , x r t/ ) 


|r , 7 x r kj \ \T kj x r H | 

,, r k , x (r A; - x r kl ) 
cos 0 - : -=— cos 9 


(15.53) 
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0 grad j ■ cos 0-grad A/ cos 0. 

r kj r kj 


15.4 Normal Mode Analysis 

The nuclear motion around an equilibrium configuration can be approximately de¬ 
scribed as the combination of independent harmonic normal modes. Equilibrium 
configurations can be found with the methods discussed in Sect. 6.2. The conver¬ 
gence is usually rather slow (Fig. 15.8) except for the full Newton-Raphson method, 
which needs the calculation and inversion of the Hessian matrix. 


15.4.1 Harmon ic Approximation 


At an equilibrium configuration 
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Fig. 15.8 (Convergence of 
energy and gradient) The 
energy of the glycine 
dipeptide is minimized with 
the methods of steepest 
descent and conjugate 
gradients 



G = C (15.54) 

the gradient of the potential energy vanishes. For small deviations from the equilib¬ 
rium 


G 


= G 




(15.55) 


approximation by a truncated Taylor series gives 

1 Q2 jj 1 

U( Ci ■■<3n) = U 0 +~Y, GO + • • • » Uo + - X H <jOCj (15.56) 

i.j ^ “ i,j 

and the equations of motion are approximately 

m iC = ~^ U = -I W uO' (15-57) 

Assuming periodic oscillations 

G = G°e“ ? (15-58) 

we have 

m,^ = Y d H u<? j - (15.59) 

j 

If mass weighted coordinates are used, defined as 


Ti 


(15.60) 
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this becomes an ordinary eigenvalue problem 


UJ 2 T: 




Hi 




,J T°. 


The eigenvectors u,- of the symmetric matrix 


H U = 


H 


ij 




m-.m : 


j 

are the solutions of 

^ \ HijUj r = A yUj r 


(15.61) 


(15.62) 


(15.63) 


and satisfy (15.61) 


Ul 2 Ui 


— ^ Hjjlijf — A, IIi, 


(15.64) 


with normal mode frequencies 

u r = J\ r . (15.65) 

Finally, the Cartesian coordinates are linear combinations of all normal modes 


o = Z c ^ eW - 

*' J / m : 


(15.66) 


Fig. 15.9 (Normal mode 
distribution for the dipeptide 
model) The cumulative 
distribution (Sect. 9.1.2) of 
normal mode frequencies is 
shown for the glycine 
dipeptide. Translations and 
rotations of the molecule 
correspond to the lowest 6 
frequencies which are close 
to zero. The highest 
frequencies between 3100 
and 3600 cm -1 correspond 
to the stretching modes of 
the 8 hydrogen atoms 
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In a true local energy minimum the Hessian matrix Hjj is positive definite and 
all frequencies are real valued. The six lowest frequencies are close to zero and 
correspond to translations and rotations of the whole system (Fig. 15.9). 


Problems 


Problem 15.1 Simulation of a Glycine Dipeptide 

In this computer experiment a glycine dipeptide (Fig. 15.5) is simulated. Parameters 
for bond stretching (Table 15.3) and bond angle (Table 15.4) terms have been derived 
from quantum calculations by Bautista and Seminario [191]. 

• Torsional potential terms (Table 15.5) can be added to make the structure more 
rigid. This is especially important for the 09 — H 17, A'4 — H 14 and N 1 — H 10 
bonds, which rotate almost freely without torsional potentials. 

• The energy can be minimized with the methods of steepest descent or conjugate 
gradients 

• A normal mode analysis can be performed (the Hessian matrix is calculated by 
numerical differentiation). The rth normal mode can be visualized by modulating 
the coordinates periodically according to 


6 - = C + c r 


: COS W r t. 


(15.67) 


• The motion of the atoms can be simulated with the Verlet method. You can stretch 
the 09 — H\1 or A'4 — H 14 bond and observe, how the excitation spreads over 
the molecule. 


Table 15.5 (Torsional potential terms) Torsional potential terms VjjM = kjjki ( 1 — cos(0,-yjt/ — 
0$H>. which can be added to the force field. Minimum angles are from the optimized structure 
without torsional terms (15.1). The barrier height of 2 kyii = 2kcal/ mo i is only a guessed value 
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Chapter 16 

Thermodynamic Systems 


An important application for computer simulations is the calculation of thermody¬ 
namic averages in an equilibrium system. We discuss two different examples: 

In the first case the classical equations of motion are solved for a system of 
particles interacting pairwise by Lennard—Jones forces (Lennard-Jones fluid). The 
thermodynamic average is taken along the trajectory, i.e. over the calculated coor¬ 
dinates at different times r ( - (t n ). We evaluate the pair distance distribution function 

= a6.D 

the velocity auto-correlation function 

C{t) = < v(t 0 )v(t) > (16.2) 

and the mean square displacement 

Ax 2 — < (x(t) — x(t 0 )) 2 > . (16.3) 

In the second case the Metropolis method is applied to a one- or two-dimensional 
system of interacting spins (Ising model). The thermodynamic average is taken over 
a set of random configurations q (,,) . We study the average magnetization 

< M >— p, < S > (16.4) 

in a magnetic field and the phase transition to the ferromagnetic state. 
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16.1 Simulation of a Lennard-Jones Fluid 

The Lennard-Jones fluid is a simple model of a realistic atomic fluid. It has been 
studied by computer simulations since Verlet’s early work [164, 199] and serves as a 
test case for the theoretical description of liquids [200, 201] and the liquid-gas [202] 
and liquid-solid phase transitions [203, 204]. 

In the following we describe a simple computer model of 125 interacting particles 1 
without internal degrees of freedom (see problems section). The force on atom i is 
given by the gradient of the pairwise Lennard-Jones potential (15.24) 






F, = S^ = -^Zv,(^-0 = 4 £ Z 


A 


12cr 12 

IJ 


j (r, - 


6o- 6 

- IT ( r '' - r /)' 


(16.5) 


We use argon parameters m — 6.69 x 10 26 kg, s = 1.654 x 10 21 J, a = 
3.405 x 10 -10 m [163]. After introduction of reduced units for length r* = 1-r, 
energy E* — \E and time t* — fejma 1 t, the potential energy 



(16.6) 


and the equation of motion 



(16.7) 


become universal expressions, i.e. there exists only one universal Lennard-Jones 
system. To reduce computer time, usually the 6-12 potential is modified at larger 
distances which can influence the simulation results [205]. In our model a simple 
cutoff of potential and forces at r max = 10A is used. 


16.1.1 Integration of the Equations of Motion 

The equations of motion are integrated with the Verlet algorithm (Sect. 13.1 1.5) 

Fj (t) , 

Ar, = r, (t) - Vj(t - At) + At 2 (16.8) 

m 

1 This small number of particles allows a graphical representation of the system during the simula¬ 

tion. 
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r, (t + At) = r,(/) + Ar, + 0(At 4 ). (16.9) 

We use a higher order expression for the velocities to improve the accuracy of the 
calculated kinetic energy 


V /+1 = 


4r, , 5F,-(f) — 2F,-(f 


At 


6 m 


At) , 

- At + 0(At 3 ). 


(16.10) 


16.1.2 Boundary Conditions and Average Pressure 

Molecular dynamics simulations often involve periodic boundary conditions to 
reduce finite size effects. Here we employ an alternative method which simulates 
a box with elastic walls. This allows us to calculate explicitly the pressure on the 
walls of the box. 

The atoms are kept in the cube by reflecting walls, i.e. whenever an atom passes 
a face of the cube, the normal component of the velocity vector is changed in sign 
(Fig. 16.1). Thus the kinetic energy is conserved but a momentum of mAv = 2mv± 
is transferred to the wall. The average momentum change per time can be interpreted 
as a force acting upon the wall 


„ Trefl 2mV -L 

Fi = < --- > . 

dt 


(16.11) 


The pressure p is given by 


P= 6& < 


j walls Z^irefl 


2mv± 


dt 


> . 


(16.12) 


With the Verlet algorithm the reflection can be realized by exchanging the values of 
the corresponding coordinate at times t„ and f„_i. 






'n+l 


Fig. 16.1 Reflecting walls 
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ill 

16.1.3 Initial Conditions and Average Temperature 


At the very beginning the N = 125 atoms are distributed over equally spaced lattice 
points within the cube. Velocities are randomly distributed according to a Gaussian 
distribution for each Cartesian component t; ;j 


/ Om) = 


Q -mvl/2k B T 


2 TTkgT 

corresponding to a Maxwell speed distribution 


/(M) = 




3/2 


4'KV 2 e~ nn ’ 2 ^ 2kBT . 


(16.13) 


(16.14) 


Assuming thermal equilibrium, the effective temperature is calculated from the 
kinetic energy 


k B T 


2 

3N Eki ‘ 


(16.15) 


The desired temperature T a is established by the rescaling procedure 


V; 


k B T 0 
k B T actua i 


(16.16) 


which is applied repeatedly during an equilibration run. The velocity distribution 
/(|u|) can be monitored. It approaches quickly a stationary Maxwell distribution 
(Fig. 16.2). 


Fig. 16.2 (Velocity 
distribution) The velocity 
distribution is shown for 
T = lOOKand T = 500K 
( histograms ) and compared 
to the Maxwell speed 
distribution (solid curves) 
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A smoother method to control temperature is the Berendsen thermostat 
algorithm [206] 


V; 


V,' 



At kTo k Tactual 


Tthe 


k T actua i 


(16.17) 


where T t h e rm is a suitable relaxation time (for instance T t herm — 20 At). This method 
can be used also during the simulation. However, it does not generate the trajectory 
of a true canonical ensemble. If this is necessary, more complicated methods have to 
be used [207] 


16.1.4 Analysis of the Results 

After an initial equilibration phase the system is simulated at constant energy (NVE 
simulation) or at constant temperature (NVT) with the Berendsen thermostat method. 
Several static and dynamic properties can be determined. 

16.1.4.1 Deviation from the Ideal Gas Behavior 

A dilute gas is approximately ideal with 


pV = Nk B T. (16.18) 

For a real gas the interaction between the particles has to be taken into account. From 
the equipartition theorem it can be found that 2 

pV = Nk B T + W (16.19) 

with the inner virial (Fig. 16.3) 

W = < 32 > F < > (16.20) 

i 

which can be expanded as a power series of the number density n — N/V [208] to 
give 


pV = Nk B T(l+b(T)y+c(T) 



(16.21) 


2 MD simulations with periodic boundary conditions use this equation to calculate the pressure. 
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Fig. 16.3 (Inner virial) The 
inner virial W (16.20, 
crosses and stars) is 
compared to pV — kgT 
(squares and circles ) for two 
values of the particle density 
N/V = 10 _3 A“ 1 (aland 
1.95 x 10 _3 A _1 (b), 
corresponding to reduced 
densities n* = a 3 N/V of 
0.040 and 0.077 



The virial coefficient b(T) can be calculated exactly for the Lennard-Jones gas [208]: 


2tt , ^ 2j~^ 2 2/ — 1 e 

b (T) = --a 3 Y J — r (^—)(—) 


072+74) 


7=0 


k„T' 


(16.22) 


For comparison we calculate the quantity 
N \Nk B T ) 


(16.23) 


which for small values of the particle density n = N/V correlates well (Fig. 16.4) 
with expression (16.22). 


Fig. 16.4 (Second virial 
coefficient) The value of 
jj — l) is shown for 

two values of the particle 
density N/ V = 10~ 3 AA 
(crosses ) and 

1.95 x 10~ 3 AA -1 (circles) 
and compared to the exact 
second virial coefficient b 
(dashed cur\>e) (16.22) 
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Fig. 16.5 (Radial pair 
distribution) The normalized 
radial distribution function 
g(R)/gideal(R) is evaluated 
for it r = 35 K, 100 K, 

1000 K and a density of 
n = 0.025 A' 3 
corresponding to a reduced 
density n* = a 3 N/V of 1.0. 
At this density the 
Lennard-Jones system 
shows a liquid-solid 
transition at a temperature of 
ca. 180 K [204] 



16.1.4.2 Structural Order 


A convenient measure for structural order [209] is the radial pair distribution function 
(Fig. 16.5) 


g(R) = < 


l 

N(N - 1) 


Y J 5(r ij -R)> = 


P(R < rij < R + clR) 

Jr 


which is usually normalized with respect to an ideal gas, for which 


(16.24) 


gideai(R) = 4 irn R 2 dR. (16.25) 

For small distances g(R)/gideai(R ) vanishes due to the strong repulsive force. 
It peaks at the distance of nearest neighbors and approaches unity at very large 
distances. In the condensed phase additional maxima appear showing the degree of 
short (liquid) and long range (solid) order. 

Equation (16.25) is not valid for our small model system without periodic bound¬ 
ary conditions. Therefore gideai was calculated numerically to normalize the results 
shown in Fig. 16.5. 


16.1.4.3 Ballistic and Diffusive Motion 


The velocity auto-correlation function (Fig. 16.6) 


C(t) = < \(t )v(f 0 ) > 


(16.26) 
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Fig. 16.6 (Velocity 
auto-correlation function) 
The Lennard-Jones system 
is simulated for 
k/jT = 200 K and different 
values of the density 
n* = 0.12(a), 0.18(b), 

0.32 (c), 0.62 (d). The 
velocity auto-correlation 
function (full curves) is 
averaged over 20 trajectories 
and fitted by an exponential 
function (dashed curves) 



0 


2 3 

time t-t Q (ps) 


4 


5 


decays as a function of the delay time t — to due to collisions of the particles. In a 
stationary state it does not depend on the initial time to. Integration leads to the mean 
square displacement (Fig. 16.6) 


Ar 2 (f) = < (x(f) - x(t 0 )) 2 > 


(16.27) 


In the absence of collisions the mean square displacement grows with (f — to) 2 , 
representing a ballistic type of motion. Collisions lead to a diffusive kind of motion 
where the mean square displacement grows only linearly with time. The transition 
between this two types of motion can be analyzed within the model of Brownian 
motion [ 210 ] where the collisions are replaced by a fluctuating random force r(t) 
and a damping constant 7 . 

The equation of motion in one dimension is 


v + jv = r(t) 


(16.28) 


with 


< r(t) > = 0 


(16.29) 



(16.30) 


m 

The velocity correlation decays exponentially 


< v(t)v{to) >= ——e 7l ' fo1 


(16.31) 


m 


with the average velocity square given by 



(16.32) 
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and the integral of the correlation function equals 

k B T 
7 m 

The average of Ax 2 is 


C(t)dt 


(16.33) 


9 2 kgT 2k b T . . 

< (jc(0 -x(t 0 )) 2 >= —- f 0 )-Va - e- 7(r -' 0) ). (16.34) 

7M7 »J7 Z 

For small time differences t — to the motion is ballistic with the thermal velocity 

< (x(t) — x(to)) 2 —— (t — to) 2 = < v 2 > (t — to) 2 . (16.35) 

m 

For large time differences diffusive motion emerges with 

< (x(t) - xito) 2 2 ^-(t - to) = 2 D(t - t 0 ) (16.36) 

in') 

with the diffusion constant given by the Einstein relation 


D = 


k B T 


(16.37) 


m 7 

For a three-dimensional simulation the Cartesian components of the position or 
velocity vector add up independently. The diffusion coefficient can be determined 
from 


1 

D = - lim 

6 f->oo 


< (x(t) - x(t 0 )) 2 > 
t — to 


or, alternatively from (16.33) [163] 


1 r°° 

D =~ < vU)v(fo) > dt. 

5 Jtr, 


(16.38) 


(16.39) 


This equation is more generally valid also outside the Brownian limit (Green-Kubo 
formula). The Brownian model represents the simulation data quite well at low parti¬ 
cle densities (Figs. 16.6 and 16.7). For higher densities the velocity auto-correlation 
function shows a very rapid decay followed by a more or less structured tail. [163, 
211,212] 
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Fig. 16.7 (Mean square 
displacement) The 
Lennard-Jones system is 
simulated for kgT = 200 K 
and different values of the 
density n* = 0.12(a),0.18(b), 
0.32 (c), 0.62 (d). The mean 
square displacement (full 
curves ) is averaged over 20 
trajectories and fitted by a 
linear function (dashed lines ) 
for t — to > 1.5ps 



16.2 Monte-Carlo Simulation 

The basic principles of Monte Carlo simulations are discussed in Chap. 9. Here 
we will apply the Metropolis algorithm to simulate the Ising model in one or two 
dimensions. The Ising model [213, 214] is primarily a model for the phase transition 
of a ferromagnetic system. However, it has further applications for instance for a 
polymer under the influence of an external force or protonation equilibria in proteins. 


16.2.1 One-Dimensional Ising Model 

We consider a chain consisting of N spins which can be either up (5,- = I) or down 
(Si = —I). The total energy in a magnetic field is (Fig. 16.8) 

N 

H = -MB = -B^nSi (16.40) 

i=l 

and the average magnetic moment of one spin is 

e iiB/kT _ g —fiB/kT g 

< M >= » eli B/kT + e -, B/ kT = Ftanh(—). (16.41) 

If interaction between neighboring spins is included, the energy of a configuration 
(.S' i • • • Sn) becomes 

N N -1 

H = —fiB ^ .S, - i^.S ; .S !+1 . 

1=1 1 = 1 


(16.42) 
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S:=±1 


J -J 


Fig. 16.8 (Ising model) N spins can be up or down. The interaction with the magnetic field is 
—pBSi, the interaction between nearest neighbors is —JS,Sj 



Fig. 16.9 (Numerical simulation of the 1 -dimensional Ising model) The average magnetization 
per spin is calculated from a MC simulation ( circles) and compared to the exact solution (16.43). 
Parameters are pB = —5 and J = —2 


The 1-dimensional model can be solved analytically [208]. In the limit (V —»■ oo the 
magnetization is 


< M > = fi 


smh(ff) 


' ; = 


(16.43) 


The numerical simulation (Fig. 16.9) starts either with the ordered state 5, = 1 or 
with a random configuration. New configurations are generated with the Metropolis 
method as follows: 
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Table 16.1 Transition probabilities for a 3-spin system (p = 1/3) 
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• flip one randomly chosen spin Si 3 and calculate the energy change due to the 
change ASi = (—Si) — Si = —2S\ 


AE — — fiBASj — J AS/(Sj-i- 1 T .S'/ 1 ) — 2 pBSj -}- 2 JSj ( 5*/_{_ 1 -}- .S ; ]) . 

(16.44) 

• if AE < 0 then accept the flip, otherwise accept it with a probability of 

p _ e -AE/kT 

As a simple example consider N=3 spins which have 8 possible configurations. The 
probabilities of the trial step are shown in Table 16.1. The table is symmetric 
and all configurations are connected. 


16.2.2 Two-Dimensional Ising Model 


For dimension d > 1 the Ising model behaves qualitatively different as a phase tran¬ 
sition appears. For B = 0 (Fig. 16.10) the 2-dimensional Ising-model with 4 nearest 
neighbors can be solved analytically [215, 216]. The magnetization disappears above 
the critical temperature T c , which is given by 


J 

kT c 


--ln(V2- 1) 
2 


1 

T27' 


Below T c the average magnetization is given by 


(16.45) 


< M > = 



_J_V. 

sinh 4 (2T )) 


(16.46) 


3 Or try one spin after the other. 
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Fig. 16.10 (Numerical 
simulation of the 
2-dimensional Ising model) 
The average magnetization 
per spin is calculated for 
6=0 from a MC simulation 
( circles ) and compared to 
(16.46) 
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Problems 

Problem 16.1 Lennard-Jones Fluid 

In this computer experiment a Lennard-Jones fluid is simulated. The pressure is cal¬ 
culated from the average transfer of momentum (16.12) and compared with expres¬ 
sion (16.19). 

• Equilibrate the system and observe how the distribution of squared velocities 
approaches a Maxwell distribution. 

• Equilibrate the system for different values of temperature and volume and inves¬ 
tigate the relation between pV/N and kT. 

• observe the radial distribution function for different values of temperature and 
densities. Try to locate phase transitions. 

• determine the decay time of the velocity correlation function and compare with the 
behavior of the mean square displacement which shows a transition from ballistic 
to diffusive motion. 


Problem 16.2 One-Dimensional Ising Model 


In this computer experiment we simulate a linear chain of N — 500 spins with peri¬ 
odic boundaries and interaction between nearest neighbors only. We go along the 
chain and try to flip one spin after the other according to the Metropolis method. 
After trying to flip the last spin .S',v the total magnetization 

N 

1=1 


(16.47) 
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Fig. 16.11 Two state model 


L 






l + 


is calculated. It is averaged over 500 such cycles and then compared graphically with 
the analytical solution for the infinite chain (16.43). Temperature and magnetic field 
can be varied. 


Problem 16.3 Two-State Model for a Polymer 

Consider a polymer (Fig. 16.11) consisting of N units which can be in two states 
Si = +1 or Si — — 1 with corresponding lengths l + and The interaction between 

neighboring units takes one of the values w ++ , w+_, w _Under the influence of 

an external force n the energy of the polymer is 



(16.48) 


This model is isomorphic to the one-dimensional Ising model. 



(16.49) 


(16.50) 



l— -|- /_)_ 

= kN—— t + Nu>. 


(16.51) 


Comparison with (16.42) shows the correspondence 


w ++ + w _— 2 w + _ 


(16.52) 


— /J,B — —n 


~ ilB = -K — -h 


/ + — /_ w ++ — w _ 


(16.53) 
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x—' /+ T* l— 

L = Y,K s i) = N ^— + 



(16.54) 


In this computer experiment we simulate a linear chain of N = 20 units with 
periodic boundaries and nearest neighbor interaction as in the previous problem. 

The fluctuations of the chain conformation are shown graphically and the mag¬ 
netization of the isomorphic Ising model is compared with the analytical expression 
for the infinite system (16.43). Temperature and magnetic field can be varied as well 
as the coupling J. For negative J the anti-ferromagnetic state becomes stable at low 
magnetic field strengths. 


Problem 16.4 Two-Dimensional Ising Model 

In this computer experiment a 200 x 200 square lattice with periodic boundaries and 
interaction with the 4 nearest neighbors is simulated. The fluctuations of the spins can 
be observed. At low temperatures ordered domains with parallel spin appear. The 
average magnetization is compared with the analytical expression for the infinite 
system (16.46). 




Chapter 17 

Random Walk and Brownian Motion 


Random walk processes are an important class of stochastic processes. They have 
many applications in physics, computer science, ecology, economics and other fields. 
A random walk [217] is a sequence of successive random steps. In this chapter we 
study Markovian [218, 219] 1 discrete time 1 2 models. In one dimension the position of 
the walker after n steps approaches a Gaussian distribution, which does not depend 
on the distribution of the single steps. This follows from the central limit theorem and 
can be checked in a computer experiment. A 3-dimensional random walk provides 
a simple statistical model for the configuration of a biopolymer, the so called freely 
jointed chain model. In a computer experiment we generate random structures and 
calculate the gyration tensor, an experimentally observable quantity, which gives 
information on the shape of a polymer. Simulation of the dynamics is simplified if the 
fixed length segments of the freely jointed chain are replaced by Hookean springs. 
This is utilized in a computer experiment to study the dependence of the polymer 
extension on an applied external force (this effect is known as entropic elasticity). 
The random motion of a heavy particle in a bath of light particles, known as Brownian 
motion, can be described by Langevin dynamics, which replace the collisions with 
the light particles by an average friction force proportional to the velocity and a 
randomly fluctuating force with zero mean and infinitely short correlation time. In a 
computer experiment we study Brownian motion in a harmonic potential. 


17.1 Markovian Discrete Time Models 

The time evolution of a system is described in terms of an /V-dimensional vector 
r(/), which can be for instance the position of a molecule in a liquid, or the price 
of a fluctuating stock. At discrete times t n — nAt the position changes suddenly 
(Fig. 17.1) 


r(Wt) = r(/„) + 2 \r„ 


(17.1) 


1 Different steps are independent. 

2 A special case of the more general continuous time random walk with a waiting time distribution 
of P(t) = S(r — At). 
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Fig. 17.1 Discrete time 
random walk 


AD 


Ar, 


r(t) 


AD 


to *1 t 2 t 3 


where the steps are distributed according to the probability distribution 1 

P{Ar n — b) =/(b). (17.2) 

The probability of reaching the position R after n + 1 steps obeys the equation 

P n +\ (R) = /J (r(t„+i) = R) 

= j <7' v b P„(R - b)/(b). (17.3) 


17.2 Random Walk in One Dimension 

Consider a random walk in one dimension. We apply the central limit theorem to 
calculate the probability distribution of the position r n after n steps. The first two 
moments and the standard deviation of the step distribution are 


b— dbbf(b ) b 2 = dbb~f(b ) <Jt, = \b 2 — b . 


-2 


(17.4) 


Hence the normalized quantity 

Ax, — b 

e = —— 

CD 


(17.5) 


3 General random walk processes are characterized by a distribution function P(R, R'). Here we 
consider only correlated processes for which P(R, R') = P(R' — R). 
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is a random variable with zero average and unit standard deviation. The distribution 
function of the new random variable 


_ 6 + 6 H-1- £« _ r„ - nb 

y/n Obyfn 

approaches a normal distribution for large n 
1 


f(v„) 




y/2n 
and finally from 

f(j n )dr n — f {rj n )drj n =f(r ]„) 


dr„ 

Ub-Jn 


we have 


(17.6) 


(17.7) 


f(r n ) 


y/2mV7b 


exp 


(r„ - nb) 2 

2 not 


(17.8) 


The position of the walker after n steps obeys approximately a Gaussian distribution 
centered at r n — nb with a standard deviation of 


fj r n — \J no i ). 


(17.9) 


17.2.1 Random Walk with Constant Step Size 

In the following we consider the classical example of a 1-dimensional random walk 
process with constant step size. At time t„ the walker takes a step of length Ax to the 

left with probability p or to the right with probability q = 1 — p (Figs. 17.2, 17.3). 

The corresponding step size distribution function is 

f(b)=p5(b + Ax) + qS(b-Ax) (17.10) 

with the first two moments 

b=(q-p)Ax tf- = Ax 2 . (17.11) 

Let the walker start at r(to) = 0. The probability P„(m) of reaching position in Ax 
after n steps obeys the recursion 


P n +\(m) = pP n (m + 1) + qP„(m - 1) 


(17.12) 
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Fig. 17.2 (Random walk with constant step size) The figure shows the position r n for three different 
1 -dimensional random walks with step size Ax = ± 1. The dashed curves show the width ±cr = 
±^/n of the Gaussian approximation (17.8) 

Fig. 17.3 Random walk 
with constant step size 


r(t)-Ax r(t) r(t)+ Ax 



which obviously leads to a binomial distribution. From the expansion of 

(p+?r = Z(")^”- m < 17 - 13 ) 

we see that 

P n {n-2m)= (j/r (17-14) 

or after substitution m' — n — 2m = — n, —n + 2 ,... n — 2, tv. 

P ’^) = ( (n _ n m > )/ 2)p (n ~ m ),2qCn+m ' )l2 ’ (17 ' 15) 

Since the steps are uncorrelated we easily find the first two moments 

n 

r n = Axj = nb = nAx(q — p ) 

i=i 


(17.16) 
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and 



n n 

^ AxjAxj = ( Axi) 2 = nb 2 = nAx 2 . 

ij =1 i=l 


(17.17) 


17.3 The Freely Jointed Chain 

We consider a simple statistical model for the conformation of a biopolymer like 
DNA or a protein. 

The polymer is modeled by a 3-dimensional chain consisting of M units with 
constant bond length and arbitrary relative orientation (Fig. 17.4). The configuration 
can be described by a point in a 3(M + 1 )-dimensional space which is reached after 
M steps Z\r, = b, of a 3-dimensional random walk with constant step size 

M 

= r 0 + ^ b,. (17.18) 

1=1 

17.3.1 Basic Statistic Properties 

The M bond vectors 

b, = r, - r, , (17.19) 

have a fixed length |b, | = b and are oriented randomly. The first two moments are 
b; = 0 b 2 = b 2 . (17.20) 

Since different units are independent 

b^ = Stjb 2 . (17.21) 


Fig. 17.4 Freely jointed 
chain with constant bond 
length b 
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Obviously the relative position of segment j 

i 

R, = r, r 0 = ^ b 

1=1 


has zero mean 

®/=i>=o 

i= 1 

and its second moment is 


R J = fe b - X b *) = X bb * = ^ 2 - 

\i=l k=l / i,k =1 

For the end to end distance (Fig. 17.5) 

M 

Rm = I’M — To = b; 

!=1 


(17.22) 


(17.23) 


(17.24) 


Fig. 17.5 (Freely jointed 
chain) The figure shows a 
random 3-dimensional 
structure with 1000 segments 
visualized as balls (Molden 
graphics [220]) 
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this gives 


R m = 0, R 2 m = Mb 2 . (17.25) 

Let us apply the central limit theorem for large M. For the x coordinate of the end to 
end vector we have 

M 

X = ^ b,e A = b ^ cos (17.26) 

!=1 i 

With the help of the averages * 4 


Y p2tT nTV 

cos 8i = — / dd> cos 9 sin 9d9 = 0 
4tt J 0 Jo 

Y p2tt i* tv 

(cos 8j) 2 = — / d(f> cos 2 9 sin ddd = - 
47r Jo Jo 3 

we find that the scaled difference 
= V3 cos 9, 

has zero mean and unit variance and therefore the sum 



converges to a normal distribution: 


P(X) = 


a/ 27r 


exp 


X 2 

"T 


Hence 


P(X) = 


1 V3 
y/2TT b\[M 


exp 


2Mb 2 


X z 


(17.27) 

(17.28) 

(17.29) 

(17.30) 

(17.31) 

(17.32) 


4 For a 1-dimensional polymer cos0; = 0 and (cos#,) 2 = 1. In two dimensions cos 6, = 

4 Jq cos ddd = 0 and (cos#;) 2 = 4 Jg cos 2 8 d8 = j. To include these cases the factor 3 in 
the exponent of (17.33) should be replaced by the dimension d. 
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and finally in 3 dimensions 
P(R m ) =P{X)P{Y)P{Z) 

Vtj 

= - , exp 

b 2 y/ (27tM) 3 


3 

2Mb 2 



17.3.2 Gyration Tensor 


For the center of mass 


Rf 


1 

M 


Z R - 


we find 

r; = o w c = 


i 

M 2 


Z RR / 


and since 


R, R, = min(i,/) b 2 


we have 



The gyration radius [221] is generally defined by 


R 


2 

9 


1 

M 


M 

Z ,R ' - R <) 2 


!=1 



fo2 M+! 

^ 1 

/M 3 

M 2 M\ 


. 1 \ 

^ Mbl 

2 

M 2 1 

l 3 

+ 2 + 6 

6 M J 

6 


(17.33) 

(17.34) 

(17.35) 

(17.36) 

Mb 2 

(17.37) 

(17.38) 

(17.39) 

(17.40) 
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R g can be also written as 




2 M 2 




(17.41) 


»'=1 7=1 


and can be experimentally measured with the help of scattering phenomena. It is 
related to the gyration tensor which is defined as 


fig 


^ X (R; - Rfj(Ri - Rr) r . 


(17.42) 


Its trace is 


tr (fig) = R] (17.43) 

and its eigenvalues give us information about the shape of the polymer (Fig. 17.6). 


17.3.3 Hookean Spring Model 

Simulation of the dynamics of the freely jointed chain is complicated by the 
constraints which are implied by the constant chain length. Much simpler is the 




^z= Q x,y Q z« n x,y 


Fig. 17.6 (Gyration tensor) The eigenvalues of the gyration tensor give information on the shape 
of the polymer. If the extension is larger (smaller) along one direction than in the perpendicular 
plane, one eigenvalue is larger (smaller) than the two other 
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o 



Fig. 17.7 Polymer model with Hookean springs 


P(b) 



-b 


b 


b 


Fig. 17.8 (Distribution of bond vectors) The bond vector distribution for a 1 -dimensional chain of 
springs has maxima at ±b. For large force constants the width of the two peaks becomes small and 
the chain of springs resembles a freely jointed chain with constant bond length 

simulation of a model which treats the segments as Hookean springs (Fig. 17.7). In 
the limit of a large force constant the two models give equivalent results. 

We assume that the segments are independent (self crossing is not avoided). Then 
for one segment the energy contribution is 



(17.44) 


If the fluctuations are small 


b,-| -b\ «* 


(17.45) 


then (Fig. 17.8) 


b; ~ b b? 


(17.46) 
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and the freely jointed chain model (17.33) gives the entropy as a function of the end 
to end vector 

s = -*• ( ' >(R "» = ‘"Gv&f) + <17 - 47) 

If one end of the polymer is fixed at ro = 0 and a force k is applied to the other end, 
the free energy is given by 


F — TS — kRm = 


bT 2 
2Mb- M 


— bR m + const. 


(17.48) 


In thermodynamic equilibrium the free energy is minimal, hence the average exten¬ 
sion is 


Rm = 


Mb 2 

- K. 

3 k B T 


(17.49) 


This linear behavior is similar to a Hookean spring with an effective force constant 


Mb 2 

3 hT 


(17.50) 


and is only valid for small forces. For large forces the freely jointed chain asymptot¬ 
ically reaches its maximum length of Rm ,max = Mb, whereas for the chain of springs 
R m -* M(b + K /f). 


17.4 Langevin Dynamics 

A heavy particle moving in a bath of much smaller and lighter particles (for 
instance atoms and molecules of the air) shows what is known as Brownian motion 
[222-224]. Due to collisions with the thermally moving bath particles it experiences 
a fluctuating force which drives the particle into a random motion. The French physi¬ 
cist Paul Langevin developed a model to describe this motion without including the 
light particles explicitly. The fluctuating force is divided into a macroscopic friction 
force proportional to the velocity 

Ffr = — 7 v (17.51) 

and a randomly fluctuating force with zero mean and infinitely short correlation time 


F randit) = 0 ¥ mnd (t )F m „ d (f) = F 2 mnd S(t - t’). 


( 17 . 52 ) 
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The equations of motion for the heavy particle are 
d 

— x = v 
dr 

A v = _ 7V + —F /r (f) - - Vt/(x) (17.53) 

dr m m 

with the macroscopic friction coefficient 7 and the potential U (x). 

The behavior of the random force can be better understood if we introduce a time 
grid r„ + i — r„ = At and take the limit At -* 0. We assume that the random force 
has a constant value during each interval 

^ rand (0 = F /; t n < t < t n + \ (17.54) 

and that the values at different intervals are uncorrelated 


F„F m = S m , n Fl 


The auto-correlation function then is given by 


F rand (OF rand(t') 


0 different intervals 
same interval. 


(17.55) 


(17.56) 


Division by At gives a sequence of functions which converges to a delta function in 
the limit At f —»■ 0 


1 - 

~7~ F rand (OF rand (0) 


F^(r-r'). 


(17.57) 


Hence we find 


— 1 - 

F = —F 

" At 


2 

rand' 


(17.58) 


Within a short time interval At —> 0 the velocity changes by 

1 1 

\{t n + At) = v — 7 \At - VU(x)At H-F n At H- (17.59) 

m m 

and taking the square gives 

2 2 jr2 

y 2 {t n + At) = \ 2 - 2 j\ 2 At - — vV U(x)At + -\F n At + ~(40 2 + • • • . (17.60) 

m m m z 


Hence for the total energy 
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E(t n + At) — —\{t n + At) + U (x(t n + At)) 

= ^v 2 (f„ + At) + U(x) + vVf/ (x)At + 


(17.61) 


we have 

p2 

E(t„ + At) — E{t n ) — m'yv 2 At + vF n At + —^( At) 2 + ■ ■ • . (17.62) 

2m 

On the average the total energy E should be constant and furthermore in d dimensions 

(17.63) 


m— d 
-v 2 = -k B T. 
2 2 


Therefore we conclude 
-=• At^ 1 


L, 


from which we obtain finally 

f 

At 


(17.64) 


(17.65) 


Problems 


Problem 17.1 Random Walk in One Dimension 

This program generates random walks with (a) fixed step length Ax = ±1 or (b) 
step length equally distributed over the interval —a/3 < Ax < -s/3. It also shows the 
variance, which for large number of walks approaches a = ~Jii. See also Fig. 17.2 

Problem 17.2 Gyration Tensor 

The program calculates random walks with M steps of length b. The bond vectors 
are generated from M random points e, on the unit sphere as b, = be,. End to end 
distance, center of gravity and gyration radius are calculated and can be averaged 
over numerous random structures. The gyration tensor (Sect. 17.3.2) is diagonalized 
and the ordered eigenvalues are averaged. 

Problem 17.3 Brownian Motion in a Harmonic Potential 

The program simulates a particle in a 1-dimensional harmonic potential 


U(x) = 



— KX 


(17.66) 
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where k is an external force. We use the improved Euler method (13.36). First the 
coordinate and the velocity at mid time are estimated 


n T o ^ 1 6 :) 


, At 

— } = x(f„) + y(t n ) — 


( At\ At F„ At f At 

I tn + — J = v(4) - 7 + — — - -x(f„) — 


m 2 m 


(17.67) 

(17.68) 


where F„ is a random number obeying (17.65). Then the values at t n+ \ are 
calculated as 


x(f„ + At) = x(t n ) + v I t„ + — I At 


('" + f) 

('• + t) 


\(t„ + At) = \(t n ) - 7 V I t„ + — ) At + — At - — x I t„ + — 


(<„+y) At. 


(17.69) 

(17.70) 


Problem 17.4 Force Extension Relation 

The program simulates a chain of springs Sect. 17.3.3 with potential energy 




(17.71) 


The force can be varied and the extension along the force direction is averaged over 
numerous time steps. 


Chapter 18 

Electrostatics 


The electrostatic potential <£(r) of a charge distribution p(r) is a solution 1 of Pois¬ 
son ’s equation 

A0(r) = —p( r) (18.1) 

which, for spatially varying dielectric constant e(r) becomes 

div(e( r) grad <£(r)) = —p( r) (18.2) 

and, if mobile charges are taken into account, like for an electrolyte or semiconductor, 
turns into the Poisson-Boltzmann equation 

div(e(r) grad <£(r)) = —pfi x ( r) — ^n°Z,ee~ z ‘ e3 ’ (r) ^ i,r . (18.3) 


In this chapter we discretize the Poisson and the linearized Poisson-Boltzmann 
equation by finite volume methods which are applicable even in case of discontin¬ 
uous e. We solve the discretized equations iteratively with the method of successive 
over-relaxation. The solvation energy of a charged sphere in a dielectric medium is 
calculated to compare the accuracy of several methods. This can be studied also in 
a computer experiment. 

Since the Green’s function is analytically available for the Poisson and Poisson- 
Boltzmann equations, alternatively the method ofboundaty elements can be applied, 
which can reduce the computer time, for instance for solvation models. A computer 
experiment simulates a point charge within a spherical cavity and calculates the 
solvation energy with the boundary element method. 


l The solution depends on the boundary conditions, which in the simplest case are given by 
li m |r]->oo ^(r) = 0. 
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18.1 Poisson Equation 

From a combination of the basic equations of electrostatics 


div D( r) = p( r) 

(18.4) 

D( r) = £(r)£(r) 

(18.5) 

E( r) = -grad ^(r) 

(18.6) 


the generalized Poisson equation is obtained 

div(e(r) grad<?(r)) = —p( r) (18.7) 

which can be written in integral form with the help of Gauss’ theorem 

® r/Adiv(e(r) grad<£(r)) = I dV e(r) gradtP(r)) = — I dV p( r). (18.8) 

JdV JV JV 

If e(r) is continuously differentiable, the product rule for differentiation gives 
£(r) A0(r) + (grad e(r)) (grad <£(r)) = —p( r) (18.9) 

which for constant e simplifies to the Poisson equation 
p(r) 

A0(r) = - — . (18.10) 

£ 


18.1.1 Homogeneous Dielectric Medium 

We begin with the simplest case of a dielectric medium with constant e and solve 
(18.10) numerically. We use a finite volume method (Sect. 12.3) which corresponds 
to a finite element method with piecewise constant test functions. The integration 
volume is divided into small cubes Vi,* which are centered at the grid points (Fig. 18.1) 

r ijk = (hi, hj, hk). (18.11) 


Integration of (18.10) over the control volume Vy k around r y k gives 


Qijk 


dV div arad 0 = 


grad <?dA = — 


dV p{ r) = 


(18.12) 
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Fig. 18.1 (Finite volume for 
the Poisson equation) The 
control volume is a small 
cube centered at a grid point 
{full circle) 



Qijk is the total charge in the control volume. The flux integral is approximated by 
(12.85) 


dV 


7 / 80 80 

grad 0 dA = -h I — (x,+i/ 2 , yj, Zk ) - (*;- 1 / 2 , yj, z k ) 


80 80 80 80 \ 

+-Q^wyj+yz,Zk) - ~^(xj,yj-i/2,zk) + —(xi,yj,zk+ i/ 2 ) - -p(xj,yj,Zk-i/2)J ■ 


(18.13) 


The derivatives are approximated by symmetric differences 


f grad 0 dA = -h {(<£(*,■+1 ,y jt Zk ) - ^(x,, y h Zk )) 

Jav 

- (0(x l , yj, Zk) - 0(Xi- U yj, Zk)) 

+ yj+u z k ) ~ yj, z k )) 

- i&ixi, yj, Zk) - 0(xt, yj-\ ,Zk)) 

+ (0(Xj, yj, Zk+i) ~ ^(xi, yj, Zk)) 

- (0(x l , yj, Zk) ~ ®(xi, yj, Zk—i ))} 

= -h ( 0(xi-i,yj, Zk) + 0(x i+ i,yj, Zk) + ■£(•*;, yj-i,Zk) + &(Xi, yj+uZk) 

+ 0(xt, yj, Zk- 1 ) + ^(Xi, yj, Zk+ 1 ) - 6 0(Xi, yj, Zk)) (18.14) 

which coincides with the simplest discretization of the second derivatives (3.40). 
Finally we obtain the discretized Poisson equation in the more compact form 


^mnjk + dr,) - 0(n jk )) = - ^ 

5=1 

which involves an average over the 6 neighboring cells 


(18.15) 


dri = {—h, 0, 0) ... dr(, — (0, 0, h). 


(18.16) 
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18.1.2 Numerical Methods for the Poisson Equation 

Equation (18.15) is a system of linear equations with very large dimension (for a 
grid with 100 x 100 x 100 points the dimension of the matrix is 10 6 x 10 6 !). Our 
computer experiments use the iterative method (Sect. 5.5) 



(18.17) 


Jacobi’s method (5.121 on p. 80) makes all the changes in one step whereas 
the Gauss-Seidel method (5.124 on p. 80) makes one change after the other. The 
chessboard (or black red method) divides the grid into two subgrids (with i+j + k 
even or odd) which are treated subsequently. The vector dr s connects points of 
different subgrids. Therefore it is not necessary to store intermediate values like for 
the Gauss-Seidel method. 

Convergence can be improved with the method of successive over-relaxation 
(SOR, 5.128 on p. 81) using a mixture of old and new values 



with the relaxation parameter w. For 1 < ui < 2 convergence is faster than for u; = 1. 
The optimum choice of lo for the Poisson problem in any dimension is discussed in 
[225], 

Convergence can be further improved by multigrid methods [226, 227]. Error 
components with short wavelengths are strongly damped during a few iterations 
whereas it takes a very large number of iterations to remove the long wavelength 
components. But here a coarser grid is sufficient and reduces computing time. After 
a few iterations a first approximation <P\ is obtained with the finite residual 


1 


(18.19) 


r\ — A0 1 + -p. 

£ 


Then more iterations on a coarser grid are made to find an approximate solution 0 2 
of the equation 


1 

A0 = —/'! =- p — A0\. 

£ 


(18.20) 


The new residual is 


r 2 — A0 2 + r,. 


(18.21) 
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Function values of 0 2 on the finer grid are obtained by interpolation and finally the 
sum 0 1 + 02 provides an improved approximation to the solution since 

1 1 

A(0 1 + # 2 ) = —P + ''1 + O 2 - r\ ) = —p + r 2 . (18.22) 

£ £ 

This method can be extended to a hierarchy of many grids. 

Alternatively, the Poisson equation can be solved non-iteratively with pseudospec- 
tral methods [228, 229]. For instance, if the boundary is the surface of a cube, eigen¬ 
functions of the Laplacian are for homogeneous boundary conditions (<£ = 0) given 
by 


yVk(r) = sin(/c v x) sin^y) sin(£-z) (18.23) 

and for no-flow boundary conditions (J^ 0 — 0) by 

N k (r) = cos(k x x) cos (k y y) cos (k z z) (18.24) 

which can be used as expansion functions for the potential 

0(r)= £ 0 k N k {r). (18.25) 

k x ,ky,k z 

Introducing collocation points r ; the condition on the residual becomes 

O = A0(Tj) + -p(Tj)= ^ k 2 0 k N k (Vj) + -p(r ; ) (18.26) 

£ K,ky,k z £ 

which can be inverted with an inverse discrete sine transformation, (respectively an 
inverse discrete cosine transformation for no-flux boundary conditions) to obtain the 
Fourier components of the potential. Another discrete sine (or cosine) transformation 
gives the potential in real space. 


18.1.3 Charged Sphere 

As a simple example we consider a sphere of radius R with a homogeneous charge 
density of 


3 


Po — e ■ 


AttR 3 


(18.27) 
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Fig. 18.2 (Discretization of the discontinuous charge density) Left the most precise method divides 
the control volumes at the boundary into two irregularly shaped parts. Middle assigning either the 
value po or zero retains the discontinuity but changes the shape of the boundary. Right averaging 
over a control volume smears out the discontinuous transition 


The exact potential is given by 

e e ( r 2 \ 

<P(r) = -b-( 1-v ) for r < R 

A-keqR StteqR \ R 2 ) 

<P(r) = - for /- > R. (18.28) 

47reor 

The charge density ( 18 .27) is discontinuous at the surface of the sphere. Integration 
over a control volume smears out this discontinuity which affects the potential values 
around the boundary (Fig. 18.2). Alternatively we could assign the value pir^) which 
is either po (18.27) or zero to each control volume which retains a sharp transition 
but changes the shape of the boundary surface and does not conserve the total charge. 
This approach was discussed in the first edition of this book in connection with a 
finite differences method. The most precise but also complicated method divides the 
control volumes at the boundary into two irregularly shaped parts [230, 231]. 

Initial guess as well as boundary values are taken from 


‘Z’o (r) = 


47T£o max(r, /?) 


(18.29) 


which provides proper boundary values but is far from the final solution inside the 
sphere. The interaction energy is given by (Sect. 18.5) 


E int = T 


p(r)0(r)dV = 


20 tteqR 


(18.30) 


Calculated potential (Fig. 18.3) and interaction energy (Figs. 18.4, 18.5) converge 
rapidly. The optimum relaxation parameter is around lo 1.9. 
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Fig. 18.3 (Electrostatic 
potential of a charged 
sphere) A charged sphere is 
simulated with radius 
R = 0.25 and a 
homogeneous charge density 
p = e ■ 3/47rff 3 . The grid 
consists of 200 3 points with 
a spacing of h = 0.025. The 
calculated potential ( circles ) 
is compared to the exact 
solution (18.28, solid cum), 
the initial guess is shown by 
the dashed line 


Fig. 18.4 (Influence of the 
relaxation parameter) The 
convergence of the 
interaction energy (18.30, 
which has a value of 
34.56 eV for this example) is 
studied as a function of the 
relaxation parameter u. The 
optimum value is around 
u R3 1.9. For uj > 2 there is 
no convergence. The dashed 
line shows the exact value 


Fig. 18.5 (Influence of grid 
size) The convergence of the 
interaction energy (18.30) 
and the central potential 
value are studied as a 
function of grid size. The 
dashed lines show the exact 
values 
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Fig. 18.6 Face center of the 
control volume 



18.1.4 Variable e 


In the framework of the finite volume method we take the average over a control 
volume to discretize e 2 and e? 


£ijk = e(r ijk ) = ^ j dV e(r) 


<Pijk = <P(Ti jk ) = - j/ dV0{ r). 

" Vijk 

Integration of (18.7) gives 


/ dV div (e(r) grad 0 (r)) = ® e(r) grad <?dA = — dV pir) 

lv Jdv Jv 


(18.31) 

(18.32) 


The surface integral is 


d 


dV 


dA e grade? = ^ / dAe{ r)— 0 . 

t — 1 1 ■ on 


-Qijk ■ 

(18.33) 


(18.34) 


sefaces " 

Applying the midpoint rule (12.77) we find (Fig. 18.6) 

6 

iA e grade? ss h 2 

Idv 


dA e grade? % lr ^ e (r ijk + ^e/r v j ^-0 + ^e/r v ^ . 


(18.35) 


2 But see Sect. 18.1.5 for the case of discontinuous e. 
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The potential 0 as well as the product e(r)j^ are continuous, therefore we make 
the approximation [230] 


( rs ‘ + \ dI ) S ( r # + \ dI ) = 


= e(r hk) 


4> (r ijk + \dr s ) - 0(r I]k ) 


e(r ijk + dr s ) 


d>{r ijk + dr s ) - 0 (rij k + 2 ^r s ) 


(18.36) 


From this equation the unknown potential value on the face of the control volume 
0(rijk + \dr s ) (Fig. 18.6) can be calculated 


\ _ e(r ijk )0(r ijk ) + e{r ijk + dr s )0(r ijk + dr s ) 
2 7 e(r ijk ) + e(r ijk + dr s ) 

which gives 


0 


rijk ■ 


(18.37) 


'{ T ‘ !k+1 2 dr )l 0 { ridk+l 2 dT ) 


2e(r,^)e(r,ji + dr s ) <P(rjj k + dr s ) - <P{rjj k ) 
e(r i jk ) + £( r iJk + dr s ) h 

(18.38) 


Finally we obtain the discretized equation 


6 

~Qijk — h ^ 

s= 1 


2£(r ijk + dr s )£(r ijk ) 
e(r ijk + dr s ) + £(r ijk ) 


(0(r ijk +dr s ) - 0(r ijk )) 


which can be solved iteratively according to 


0 new (r ijk ) = 


Z 


2s(,rijt+dr s )s(Tiit) 

e(r ijk +dr s )+E(Xij k ) 


0 old (r ijk + dr s ) + 


Qijk 

li 


Z 2e(r,j t +4r,')£(r,jt) 
E(r ijk +dr s )+E(r iJk ) 


(18.39) 


(18.40) 


18.1.5 Discontinuous e 

For practical applications models are often used with piecewise constant e. A simple 
example is the solvation of a charged molecule in a dielectric medium (Fig. 18.9). 
Here e = eo within the molecule and e = £()S| within the medium. At the boundary 
e is discontinuous. In (18.40) the discontinuity is replaced by a smooth transition 
between the two values of £ (Fig. 18.7). 

If the discontinuity of e is inside a control volume Vy k then (18.31) takes the 
arithmetic average 
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Fig. 18.7 (Transition of e) 
The discontinuous e(r) 

(black line ) is averaged over 
the control volumes to obtain 
the discretized values eyk 
(full circles). 

Equation (18.40) takes the 
harmonic average over two 
neighbor cells (open circles) 
and replaces the 
discontinuity by a smooth 
transition over a distance of 
about h 



Fig. 18.8 Average of e over 
a control volume 


e l 

Vi 

e 2 

CM 

> 


Vi 

V 2 

e i 

E 2 



e ijk = V^ £l + V$e 2 (18.41) 

which corresponds to the parallel connection of two capacities (Fig. 18.8). Depending 
on geometry, a serial connection may be more appropriate which corresponds to the 
weighted harmonic average 


£ ijk 


V (1) £ -1 4- V (: 

v£ 1 + v iii 


( 2 ) -1 ' 


r ijk 


ijk 


(18.42) 


18.1.6 Solvation Energy of a Charged Sphere 

We consider again a charged sphere, which is now embedded in a dielectric medium 
(Fig. 18.9) with relative dielectric constant £\. 

For a spherically symmetrical problem (18.7) can be solved by application of 
Gauss’s theorem 
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Fig. 18.9 (Solvation of a 
charged sphere in a dielectric 
medium) Charge density and 
dielectric constant are 
discontinuous at the surface 
of the sphere 



d0 


4nr z e(r) - = —47T / p(r')r' 2 dr' = —q(r) 

dr l n 


0(0 = - 


q(r ) 


/o 47rr 2 s(r) 

For the charged sphere we find 


+ 0 ( 0 ). 


q(r) = 

0(f) = 
0(0 = - 


Qr 2 /R 3 for r < R 
Q for r > R 

Q r 2 


4tt£qR 2 2 

Q 


+ 0(0) for r<R 

Q 


0 ( 0 ) 


e+ 


-I for r > R. 


StteoR 47T£o£i 

The constant 0(0) is chosen to give vanishing potential at infinity 

Q Q 

0 ( 0 ) = —--+ 

4tT£()£iR 8tt£qR 

The interaction energy is 


_ 1 
Eint — _ 


4irr~dr p<P(r) = 


Q 2 ( 5 + et) 

407T£ 0 ei^ 


Numerical results for £\ = 4 are shown in Fig. 18.10. 


(18.43) 

(18.44) 

(18.45) 

(18.46) 

(18.47) 

(18.48) 

(18.49) 


18.1.7 The Shifted Grid Method 

An alternative approach uses a different grid for e which is shifted by h /2 in all 
directions (Fig. 18.11) [232] or, more generally, a dual grid (12.74). 


~ijk — ^(Xi+ i /z t J+ 1 /2,k+ 1 /2). 


(18.50) 
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Fig. 18.10 (Charged sphere 
in a dielectric medium) 
Numerical results for £i = 4 
outside the sphere and 200 3 
grid points ( circles ) are 
compared to the exact 
solution (18.46,18.47, solid 
curves ) 



Fig. 18.11 (Shifted grid 
method) A different grid is 
used for the discretization of 
e which is shifted by h/2 in 
all directions 



The value of e has to be averaged over four neighboring cells to obtain the discretized 
equation 

Qiiic x ’ d<P 

- = ^e(r ijk + dr s ) — (r ijk + dr s ) 

S 

_ ^ij,k +1 — ^ij.k £ ijk + £ ij-l,k + £ i-lj,k + s i-lJ-l,k 

~ h 4 

d^jjk £ ijk— 1 + £ i,j—\,k—l T £ i—lj,k—l T £ i—l,j—l,k—l 

h 4 

®i +! ,j.k d^ijk Eij k T EiJ—l t k T £ i,j,k— 1 T £ i,j —\,£—1 

it 4 
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Fig. 18.12 (Comparison of 
numerical errors) The 
Coulomb interaction of a 
charged sphere is calculated 
with several methods for 
100 3 grid points, circles 
(18.40, s averaged) 
diamonds (18.40, e~ l 
averaged) squares (18.51, 
e averaged), triangles (18.51, 
e _ 1 averaged), solid curve 
analytical solution (18.49) 



‘t’i-lj.k — G’iJJ £ i-ljk + £ i-\,j-\,k + £ i-\J,k-l + £ i-\,j-\,k-\ 

h 4 

*&ij+l,k ®i,j,k -ijk T" £ i— I J,k T" I T" £ i— I J,k— I 

+ h 4 

^i,j-\,k ~ ‘Pi.j.k £ ij-\k + £ i—lj—l,k + £ i,j-\,k-\ + Ej-lj-l^fc-l 

ll 4 


(18.51) 


The shifted-grid method is especially useful if e changes at planar interfaces. 
Numerical results of several methods are compared in Fig. 18.12. 


18.2 Poisson-Boltzmann Equation 

Electrostatic interactions are very important in molecular physics. Bio-molecules 
are usually embedded in an environment which is polarizable and contains mobile 
charges ( Na + , K + , Mg ++ , Cl~ ■ ■ ■). 

We divide the charge density formally into a fixed and a mobile part 

PiV') — Pfix (f ) + PmobileiV) • (18.52) 

The fixed part represents, for instance, the charge distribution of a protein molecule 
which, neglecting polarization effects, is a given quantity and provides the inhomo¬ 
geneity of the equation. The mobile part, on the other hand, represents the sum of all 
mobile charges (e is the elementary charge and Z, the charge number of ion species i) 


PmobileiV ) — ^ . ZjC tlj (l') 


(18.53) 
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which move around until an equilibrium is reached which is determined by the mutual 
interaction of the ions. The famous Debye-Huckel [233] and Gouy-Chapman models 
[234, 235] assume that the electrostatic interaction 

U (r) = Z,e<Z>(r) (18.54) 

is dominant and the density of the ions n, is given by a Boltzmann-distribution 

m( r) = „fV z ^(r VKbT. (18 .55) 

The potential r) has to be calculated in a self consistent way together with the 
density of mobile charges. The charge density of the free ions is 

Pmobile (r) = X nfeZ ie - z ‘ e<f/kBT (18.56) 

i 

and the Poisson equation (18.7) turns into the Poisson-Boltzmann equation [236] 
div( £ (r) grad<5(r)) + ^ nf ) eZ i e- x “‘ p / k * T = - Pfix (r). (18.57) 


18.2.1 Linearization of the Poisson-Boltzmann Equation 


For small ion concentrations the exponential can be expanded 



For a neutral system 


x»r^= o 

i 

and the linearized Poisson-Boltzmann-equation is obtained: 

Z 2 e 2 

div(e(r) grad <P( r)) - V ;?, (0> 0(r) = -p fix . 

KbI 

i 

With 

£(r) = e 0 £ r (r) 


(18.58) 


(18.59) 


(18.60) 


(18.61) 


and the definition 
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K(r) 2 


—-—y4 0) zf 

£ 0 £ r (r)k B T ^ 


we have finally 


(18.62) 


9 1 

div(£,.(r) arad 0( r)) — e r K 0 =- p. 

£o 


(18.63) 


For a charged sphere with radius a embedded in a homogeneous medium the solution 
of (18.63) is given by 


0 



A = 


47T£o£ r 1 + na 


(18.64) 


The potential is shielded by the ions. Its range is of the order A Debye = 1/k (the 
so-called Debye length). 


18.2.2 Discretization of the Linearized Poisson Boltzmann 
Equation 


To solve (18.63) the discrete equation (18.39) is generalized to [237] 

2,€ r (Xijk T (Xijk)) 


^ eAr 


£r(r,y* + dr s ) + £ r (r ijk) ) 


(&(r ijk + dr s ) - 0(r ljk )) 


- £ r (r, ; OK 2 (r i: ,D/r 0(r ljk ) = 


If £ is constant then we iterate 


Qijk 
he o 


(18.65) 


^+Z^0r ijt+drj 

6 + h 2 K 2 { r ijk ) 


0 new (r ijk ) = 


(18.66) 


18.3 Boundary Element Method for the Poisson Equation 

Often continuum models are used to describe the solvation of a subsystem which is 
treated with a high accuracy method. The polarization of the surrounding solvent or 
protein is described by its dielectric constant £ and the subsystem is placed inside a 
cavity with e = £q (Fig. 18.13). Instead of solving the Poisson equation for a large 
solvent volume another kind of method is often used which replaces the polarization 
of the medium by a distribution of charges over the boundary surface. 
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Fig. 18.13 Cavity in a 



dielectric medium 


e = e 


In the following we consider model systems which are composed of two spatial 
regions: 

• the outer region is filled with a dielectric medium (e\ ) and contains no free charges 

• the inner region (“Cavity”) contains a charge distribution p(r) and its dielectric 
constant is e = eg. 


18.3.1 Integral Equations for the Potential 

Starting from the Poisson equation 


div(e(r)grad0(r)) = —p( r) 


(18.67) 


we will derive some useful integral equations in the following. First we apply Gauss’s 
theorem to the expression [150] 

div [G(r — r , )E(r)grad(tf>(r)) — 4>(r)e(r)grad(G(r — r'))] 

= —p(r)G(r — r') — <f>(r)e(r)divgrad(G(r — r')) — <?(r)grade(r)grad(G(r — r')) 


(18.68) 


with the yet undetermined function G(r — r'). Integration over a volume V gives 



— dV (p(r)G(r — r') + 0 (r)e(r)divgrad(G(r — r')) 


v 


+0(r) grade(r)grad(G(r - r'))) 



Now choose G as the fundamental solution of the Poisson equation 
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Fig. 18.14 Discontinuity at 
the cavity boundary 


G 0 (r-r') = — 


1 


47r|r — r' 

which obeys 

div gradGo = <5(r — r') 



(18.71) 


to obtain the following integral equation for the potential: 



dV ^ (r) + [ dV 0 (r ) grade (r) grad (-— 1 —\ 

47r|r — r'| 4n J v \|r-r'|/ 

1 d d ( 1 \\ 

t --e(r) — (0(r)) + <2>(r)e(r) — - -- . (18.72) 

|r — r'| on on \ |r —r'|// 


First consider as the integration volume a sphere with increasing radius. Then the 
surface integral vanishes for infinite radius (0 —> 0 at large distances) [150]. 

The gradient of e(r) is nonzero only on the boundary surface (Fig. 18.14) of the 
cavity and with the limiting procedure (d 0) 


grade {r)dV = n 


£i - 1 

d 


e 0 dV = dA n(ei — l)eo 


we obtain 


0{r') = 


1 /’ p( r) 

- / dV ——— 

e(r') J cav 47r|r — r'| 


+ 


(ei - l)eo 
47re(r') 


dA 0 (r) 


d_ _1_ 

dn |r — r'| 


(18.73) 


This equation allows to calculate the potential inside and outside the cavity from 
the given charge density and the potential at the boundary. 

Next we apply (18.72) to the cavity volume (where e = eo) and obtain 


<Z>;„( r') 



Pi r) 

47r|r - r'|e 0 


1 

47T 



d_ _1_ 

dn |r — r'| 


_J._ d_ 

|r — r'| dn 



(18.74) 
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From comparison with (18.73) we have 


1 d { 

dA- -<£,«(r) = ei ® dA<P in { r) 


Ir — r'l dn 


Js 


d 1 

dn Ir — r' 


and the potential can be alternatively calculated from the values of its normal gradient 
at the boundary 


0(r f ) = 


1 


e(r') 


dV 


p(x) 




47r|r — r'| 47re(r') J s |r — r'| dn 


1 d 

dA- --—‘Mr). 


(18.75) 


This equation can be interpreted as the potential generated by the charge density p 
plus an additional surface charge density 


cr(r) = 


- 0-3 


d 


- 1 


(18.76) 


Integration over the volume outside the cavity (where e = eieo) gives the following 
expression for the potential: 


0 o ^=ii dA { 


®out{ r)- 


d 1 


1 d 


dn |r — r'| |r — r'| dn 
At the boundary the potential is continuous 
&out(r) = &i„(r) r e .4 


&out(r) 


(18.77) 


(18.78) 


whereas the normal derivative (hence the normal component of the electric field) has 
a discontinuity 


d&n 


er 


dn 


d&h, 

dn 


(18.79) 


18.3.2 Calculation of the Boundary Potential 

For a numerical treatment the boundary surface is approximated by a finite set of 
small surface elements S,, i = 1 • ■ • N centered at r, with an area A, and normal 
vector n,- (Fig. 18.15). (We assume planar elements in the following, the curvature 
leads to higher order corrections). 

The corresponding values of the potential and its normal derivative are denoted 
as 0j — 0(r,) and = n, grade? 1 (r,). At a point r* close to the element Sj we 
obtain the following approximate equations: 
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Fig. 18.15 Representation 
of the boundary by surface 
elements 



f p( r) 

_iy*i a A_L + l 

47r Js. dn |r — r. I 47 t dn 

i 1 J i 

1 ^ f d 1 1 ^ 

4, “' ,r A = s I>£ " jir-F - s ? 


r/A 


1 


r - r,. 


(18.80) 


Si - -7 

d0,, 


dn 


dA 


l r - r 7 


(18.81) 


These two equations can be combined to obtain a system of equations for the potential 
values only. To that end we approach the boundary symmetrically with r f = r, ±c/n,. 
Under this circumstance 


dA 


1 


dA 


|r - r; | 
d 1 

dn |r — r+ 

d 1 

dA —- T 

Si dn |r - rj 


= j) d A 


1 


r *7 


= -<b dA 


d 1 
dn lr — r7 


= <b d A 


d 1 


dn |r — r 


Z- j*i 


(18.82) 


and we hnd 

(1 + £\)0j = [ dV 


P( r) 


47re 0 |r - r,- 


1 ^ f d 1 1 /■ 0 1 

— V(i -d )0i & dA-- -— - — (1 +ei )0j (h dA-- -- 

4tt J s . dn |r — r ; | 4?r J Sj dn |r - r ; 


The integrals for i ^ j can be approximated by 


(18.83) 


dA 


d_ _1_ 

dn lr — r7 


= Ai n/grad, - 


(18.84) 


The second integral has a simple geometrical interpretation (Fig. 18.16). 
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Fig. 18.16 Projection of the 
surface element 



Since gradyy^ = — jj—jjj the area element dA is projected onto a sphere 
with unit radius. The integral f s dA grad r _ r ^ r is given by the solid angle of S y 

with respect to r. For r — > /y from inside this is just minus half of the full space 
angle of 4n. Thus we have 


(1 + ex)<Pj = 



jg(r) 

47r|r - r,-1e 0 


1 

4-7T 


Xd - ^ 0iAi 


d 1 


dm |r ; - 


+ ^(l+ei )4>j 


(18.85) 


or 


*j = 


2 

1 + et 



P( r) 

47re 0 |r - ry| 



ci - 1 . d 1 

- <PjAi -. 

et + 1 dm |r,- - ryl 


(18.86) 


This system of equations can be used to calculate the potential on the boundary. The 
potential inside the cavity is then given by ( 1 8.73). Numerical stability is improved by 
a related method which considers the potential gradient along the boundary. Taking 
the normal derivative 


d_ 

dnj 


n fgrad r .± 


(18.87) 


of (18.80, 18.81) gives 


d_ 

dnj 


in (ly ) 


d_ 

dnj 


f dV P,r) 

Jv 47T|r - Tj |e 0 


LyqJciaJ^ - 1 _ + { dA JL_L_ 

47T •“ J s . dndry |r — iy | 47r •“ dn J Sj dnj |r — iy \ 


(18.88) 
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d 1 


& 2 


s . dndrij |r — r+ 


1 ^ 80 

4-7T ' 


dn 


dA ■ 

s, dnj |r-r+| 


(18.89) 


In addition to (18.82) we have now 


dA 


8 2 


Sj dndrij |r — r. 


1 / 8 2 
= i d a 


s . dndnj |r — r+ 


(18.90) 


and the sum of the two equations gives 


K) 


1 8n 0mj 


= dV 


P(r) 


+ 


dn 

1 + if d0 JJn 


47re 0 |r - r,- 


+ 


1 - 2 . 
_£l_ 

47T 


Z^ 


<9<Z>, 


dn |r f — r 7 -| 


27 t dn 

or finally 


(18.91) 


P( r) 


d 2a d f 

T, G’inJ = ~T7 / dV , , 

dnj e\ + 1 diij J v 47re 0 |r - r 


6] — 1 x—' 

+2 — 2> 


£1 + 


>W 


d0j,m d 1 
dn dnj\ri—rj\ 


(18.92) 


In terms of the surface charge density this reads: 


°j = 2£( > 


(1 — £ i) 
(1 + ei) 



dV 


p(r) 

47reo|r — r'| 


+ 


1 

47T£o 


Z a i A ‘ 

iAj 


n /( r / ~ r/) \ 

l r / - r,-! 3 j 


(18.93) 


This system of linear equations can be solved directly or iteratively (a simple damping 
scheme a' m —*■ uxj' m + (1 — u))<r' m old with u> « 0.6 helps to get rid of oscillations). 
From the surface charges <t,A; the potential is obtained with the help of (18.75). 
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18.4 Boundary Element Method for the Linearized 
Poisson-Boltzmann Equation 


We consider now a cavity within an electrolyte. The fundamental solution of the 
linear Poisson-Boltzmann equation (18.63) 


G k (r - r') = 


e 


—re|r—r'| 


(18.94) 


47r|r — r'| 

obeys 

div gradGJr - r') - k 2 G k (r - r') = <5(r - r'). (18.95) 

Inserting into Green’s theorem (18.69) we obtain the potential outside the cavity 

®out{ r') = ~f s dA (®our(r)~G K (r - r') - GJr - r')^& out (r)\ (18.96) 

which can be combined with (18.74, 18.79) to give the following equations [238] 


(1 +ei)<P(r') =fdA 


d d 

^(r)^-(Go — eiGJ — (Go — G K ) —0 m (r) 
on on 


+ 


P( r) 


47re ( )|r — r' 


-dV 


1 cav 


d f ^2 

(1 + e,) 4> ln {r') = dA0(r)—— (G 0 - GJ 

on J s anon 


J d d 


- (go - 4c») 


p( r) 


d 


dn' J cav 4?re|r - r'| 


dV. 


(18.97) 


(18.98) 


For a set of discrete boundary elements the following equations determine the values 
of the potential and its normal derivative at the boundary: 

1 + ei x - / d x ’ d f 

—-— 0j = 2_j <d>i f dA ~Q^(Go — £t G k ) — f dA(Go — G K ) 


+ 


•AJ 
P( r) 

47re 0 |r - r, 
1 + ei d 
2 dn’ 


m 


dn 


-dV 


(18.99) 


-W 


d 2 




iAi 


P( r) 


d 


dn' J 47re|r — r, 


dV. (18.100) 
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The situation is much more involved than for the simpler Poisson equation (with 
k = 0) since the calculation of many integrals including such with singularities is 
necessary [238, 239]. 


18.5 Electrostatic Interaction Energy (Onsager Model) 

A very important quantity in molecular physics is the electrostatic interaction of a 
molecule and the surrounding solvent [240, 241]. We calculate it by taking a small 
part of the charge distribution from infinite distance (<P(r —> oo) = 0) into the 
cavity. The charge distribution thereby changes from A p(r) to (A + dX)p(r) with 
0 < A < 1. The corresponding energy change is 


dE = J dX ■ p(r) @\(r)dV 


Xp(r') 


47T £ 0 |r 



dV. 


(18.101) 


Multiplication of the equations (18.93) by a factor of A shows that the surface 
charges Aer„ are the solution corresponding to the charge density A p(r). It follows 
that <t„(A) = Act,, and hence 


dE = XdX / p(r) 


( S 


+ 


p(r') 


47T£ 0 |r — r n \ 47T£ 0 |/‘ — r'\ 


r)- 


(18.102) 


The second summand is the self energy of the charge distribution which does not 
depend on the medium. The hist summand vanishes without a polarizable medium 
and gives the interaction energy. Hence we have the final expression 


E int = fdE= f XdX [ p(r) T „ a f n - -dV 

J Jo J „ 47T£ok - r„\ 




P(r) 


87T£ 0 |/- - r„ 


-dV. 


(18.103) 


For the special case of a spherical cavity with radius a an analytical solution by a 
multipole expansion is available [242] 
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87T£q 


zz 
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[/ + £!(/+ l)]fl 2 '+‘ 




(18.104) 
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with the multipole moments 
M'r = [ p(r , 8, ip) 


47r 


r l Y™(8, <p)dV. 


21+ 1 

The first two terms of this series are: 
1 ei - 1 


F (ff) = 

^int 


-M"M" = 


Q 2 


p (1) _ 

int 


8tt£o ei a 

1 2(ei - 1) 
87teo (1 + 2£i)a 3 
1 2(£ 1 -l) A t 2 

87teo 1 + 2ei a 3 


o = _L_ /1_ 

87T£ 0 V £\) 


(Mf *Mf 1 + M° X M" + M\m\) 


(18.105) 


(18.106) 


(18.107) 


18.5.1 Example: Point Charge in a Spherical Cavity 


Consider a point charge Q in the center of a spherical cavity of radius R (Fig. 18.17). 
The dielectric constant is given by 


£o r < R 
£i£o r > R 


(18.108) 


Electric field and potential are inside the cavity 


E = 


Q 


4ne 0 r 2 


0 = 


Q 


Q 


47 T£ 0 r 4 tt£ 0 R 


(H 


and outside 


(18.109) 


E = 


Q 


47T£i£or 2 


0 = 


Q 


47T£i£or 


r > R 



(18.110) 
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Fig. 18.17 Surface charges 
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Fig. 18.18 (Solvation 
energy with the boundary 
element method) A spherical 
cavity is simulated with 
radius a = 1A which 
contains a point charge in its 
center. The solvation energy 
is calculated with 25 x 25 
( circles ) and 50 x 50 
(squares) surface elements of 
equal size. The exact 
expression (18.106) is shown 
by the solid curve 



which in terms of the surface charge density a is 


Q + 4n Fra 

E = - -— r > R 

4ireor- 

with the total surface charge 


(H- 


4irR~cr = Q ( — 

\ £ i 

The solvation energy (18.103) is given by 

Q 2 


(18.111) 


(18.112) 


Ejnt — 


87T£o 


(H 


(18.113) 


which is the first term (18.106) of the multipole expansion. Figure 18.18 shows 
numerical results. 


Problems 

Problem 18.1 Linearized Poisson-Boltzmann Equation 

This computer experiment simulates a homogeneously charged sphere in a dielec¬ 
tric medium (Fig. 18.19). The electrostatic potential is calculated from the linearized 
Poisson Boltzmann equation (18.65) on a cubic grid of up to 100 3 points. The poten¬ 
tial 0 ( x ) is shown along a line through the center together with a log-log plot of the 
maximum change per iteration 
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Fig. 18.19 Charged sphere 
in a dielectric medium 


Fig. 18.20 Point charge 
inside a spherical cavity 


\0 in+l) (r) - <2> (n) (r)l 



as a measure of convergence. 

Explore the dependence of convergence on 

• the initial values which can be chosen either 0(r) = 0 or from the analytical 
solution 


0(r) 


Q 2+e(l +Ka) 

87re€otf 1 +Ka 

Q e -K(r-a) 

4neoe(Ka+l)r 


S—r 2 

87T6o 


for r < a 


for r > a. 


(18.115) 


• the relaxation parameter u for different combinations of e and k 

• the resolution of the grid 

Problem 18.2 Boundary Element Method 

In this computer experiment the solvation energy of a point charge within a spherical 
cavity (Fig. 18.20) is calculated with the boundary element method (18.93). 
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The calculated solvation energy is compared to the analytical value from (18.104) 


Esolv - 


Q 2 y s 2n (et-e 2 )(n+l) 
87reo R , R 2n ne i + (n + l)e-> 

n= 1 


(18.116) 


where R is the cavity radius and s is the distance of the charge from the center of the 
cavity. 

Explore the dependence of accuracy and convergence on 

• the damping parameter w 

• the number of surface elements (6 x 6 ■ ■ • 42 x 42) which can be chosen either as 
dcjidd or d(j)d cos 0 (equal areas) 

• the position of the charge 




Chapter 19 

Advection 


Transport processes are very important in physics and engineering sciences. Trans¬ 
port of a conserved quantity like energy or concentration of a certain substance 
(e.g. salt) in a moving fluid is due to the effects of diffusion (Chap. 21) and advection 
(which denotes transport by the bulk motion). The combination of these two transport 
mechanisms is usually called convection. 

In this chapter we investigate the advection equation in one spatial dimension 

d d 

-f(x,t) = -c—f(x,t). (19.1) 

ot ox 

Numerical solutions are obtained with simple and more elaborate methods using 
finite differences, finite volumes andfinite elements. Accuracy and stability of different 
methods are compared. The linear advection equation is an ideal test case but the 
methods are also useful for general nonlinear advection equations including the 
famous system of Navier-Stokes equations. 


19.1 The Advection Equation 


Consider a fluid moving with velocity u(r) and let /(r, t) denote the concentration 
of the substance. Its time dependence obeys the conservation law 


d_ 

di 


f = div ( D grad / - u/) + S(r, t) = - div (J diff + J adv ) + S( r, t) 


(19.2) 


or in integral form 


[ dVf(r,t)+(b J(r, t)clA = I dVS(r,t). 


dt 


(19.3) 


dV 


Without diffusion and sources or sinks, the flux of the substance is given by 
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Fig. 19.1 Advection in an 
incompressible fluid 


J(r, t) = u(r, t)f (r, r) 



and the continuity equation for the substance concentration reads 

jrf + div (/ u) = 0 . 
at 

Introducing the substantial derivative we obtain 

d d 

0 = —/ + div (/ u) = —/ + (u grad) / + / divu 


(19.5) 


(19.6) 


d/ 

dr 


+ / divu. 


(19.7) 


For the common case of an incompressible fluid div u = 0 and the advection equation 
simplifies to 


d/ 

dr 


d_ 

dt 


/(r, r) + (u(r, r) grad) /( r, r) = 0 


(19.8) 


which has a very simple interpretation. Consider a small element of the fluid 
(Fig. 19.1), which during a time interval At moves from the position r to r + Ar = 
r + uAt. The amount of substance does not change and we find 


df 

/(r, r) = /(r + u At, t + At) = r, r) + ——At + u At grad / ■ 

at 


(19.9) 


which in the limit of small At becomes (19.8). 


19.2 Advection in One Dimension 

In one dimension div u = 717=0 implies constant velocity u x = c. The differential 
equation 
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can be solved exactly with d’ Alembert’s method. After substitution 
x' = x — ct t' = t 
f(x, t) = fix' + Ct', t') = fix', t') 

d d d d d 

dx dx' dt dt' ° dx' 

it becomes 

„ ( d d d\ d J 

_ v* 7 - C d^ + C d7') ^ 

hence f does not depend on time and the solution has the 
fix,t) = fix') = fix - ct) 

where the constant envelope is determined by the initial values 
fix') = fix, t = 0). 


(19.11) 


(19.12) 


(19.13) 


(19.14) 


(19.15) 


After spatial Fourier transformation 


fik, t) 



a ikx 


fix, t)dx 


(19.16) 


the advection equation becomes an ordinary differential equation 


d fit, k) 
dr 


Akx 


~J2tt J-o 


i ck fix, t)dx — i ck fit, k) 


(19.17) 


quite similar to the example of a simple rotation (p. 13). Therefore we have to expect 
similar problems with the simple Euler integration methods (p. 293). 

For a Fourier component of / in space and time (i.e. a plane wave moving in 
x-direction) 


9uk = 


e i (ut-kx) 


(19.18) 


we find a linear dispersion relation, i.e. all Fourier components move with the same 
velocity 


U! = ck. 


(19.19) 
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19.2.1 Spatial Discretization with Finite Differences 

The simplest discretization (p. 259) is obtained by introducing a regular grid 

x m — mAx m = 1,2... M (19.20) 

fn(t)= (19.21) 


and approximating the gradient by a bnite difference quotient. 

In the following we use periodic boundary conditions by setting /> = f M , f M+ \ = 
fi which are simplest to discuss and allow us to simulate longer times on a finite 
domain. 


19.2.1.1 First Order Forward and Backward Differences (Upwind 
Scheme) 

First we use a first order backward difference in space 

d f„(t) f m (t) - 1(0 

-= c -. 

dr Ax 

From a Taylor series expansion 


AS S, S d f A , 

f(x - Ax) = f{x) - H- 2 — ~df2 


■ ■ — exp 


d 

ox 


fix) 


(19.23) 


we see that the leading error of the finite difference approximation 

df fix) - fix - Ax) _ df df Ax d 2 f 
dt Ax dt dx 2 dx 2 


(19.24) 


looks like a diffusion term for positive velocity c and is therefore called “numeri¬ 
cal diffusion”. Negative velocities, instead lead to an unphysical sharpening of the 
function f. 

For c < 0 we have to reverse the space direction and use a forward difference 


d/m it) fm+lit) - fmit) 

-= c - 

dr Ax 


for which the sign of the second derivative changes 
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df__ fix + Ax) - f(x) _ Of dj__ Ax d 2 f 
dt Ax dt C dx C 2 dx 2 


(19.26) 


Using the backward difference we obtain a system of ordinary differential 
equations 


d 

dr 


/ /i \ 
h 

fM—\ 

\ fM J 


( 1 


C 

Ax 


1 1 


-1\ / /i \ 

h 


-l i 


■i i / 


/m-i 

\ /m / 


(19.27) 


or shorter 


df c 

— =-Mf 

dr Ax 


(19.28) 


with the formal solution 


f(r) = exp | — M t J f(r = 0). 


(19.29) 


The eigenpairs of M are easily found (see p. 221). Inserting the Ansatz 


fk = 


/ 


V e 


-ikAx 


-MikAx 


(19.30) 


corresponding to a Fourier component (19.18) into the eigenvalue equation we obtain 


Mf k = 


e -(M-l)ikAx _ t -(M-2)ikAx 


\ e 


-MikAx 


r±—(M—l)ikAx 




Solutions are found for values of k given by 


MikAx _ l k = Qj 


27T 


. (M — 1)- 


27T 


MAx MAx 

or, reducing k-values to the first Brillouin zone (p. 132) 

27t 27t 27t M 'll r 

, 0 , 


-(?-) 


MAx MAx 


MAx 


2 MAx 


M even 


(19.31) 


(19.32) 


(19.33) 
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k 


\1 2tt 
T MAx 


2ir 2 tt M 2i r 

-, 0 ,-- 

MAx MAx 2 MAx 


M odd 


for which 


(19.34) 


Aff k = A,f, = (1 — e' kAx ) f k . 

The eigenvalues of — M are complex valued 


Uk = -—(1 — e lkAx ) — - (coskAx — 1) + i— sin kAx 

Ax Ax Ax 

and so is the dispersion 

c c 

u> k = — i(T k = — sin kAx — i — ( coskAx — 1). 

Ax Ax 

If we take instead the forward difference we find similarly 


Ok = -(e lkAx — 1) =-(cos kAx — 1) + i— sinkAr 

Ax Ax Ax 


c c 

ui k = ~i\ k = — sin kAx + i — (cos kAx — 1). 
Ax Ax 


(19.35) 

(19.36) 

(19.37) 

(19.38) 

(19.39) 


19.2.1.2 Second Order Symmetric Difference 

A symmetric difference quotient has higher error order and no diffusion term 


f(x + Ax) - f(x - Ax) _ sinh (Ax£) df (Ax) 2 d 3 f 

2Ax Ax dx 6 <9x 3 


(19.40) 


It provides the system of ordinary differential equations. 


d 

dr 


/ f' \ 

h 

/m- 1 

V /m / 


c 

Ax 


( 0 1/2 

- 1/2 0 1/2 


-V 2 \ / /. \ 

h 


\ */2 


- 1/2 0 1/2 

-V2 0 / 


/m-i 

V /m / 


(19.41) 


The eigenpairs of M are easily found (see p. 221) from 
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0 0.2 0.4 0.6 0.8 1 

k Ax/lt 


<1 

3 

s' 


Fig. 19.2 (Dispersion of finite difference quotients) The dispersion of first order (19.37, 19.39) 
and second order (19.45) difference quotients is shown. Left the real part of uj k (full curve) which 
is the same in all three cases is compared to the linear dispersion of the exact solution (broken line). 
Right imaginary part of a i k (dashed curve = forward difference, dash-dotted curve = backward 
difference, full line = second order symmetric difference) 


M f k = 


1 

2 


/ g—2i kAx Q—MikAx 

^—3ikAx _ Q—ikAx 


g—MikAx _ Q—(M—2)ikAx 

i kAx Q—(M—l)ikAx 


(19.42) 


= l(e-^. e ^)f 4 = —isin^f* (19.43) 

hence the eigenvalues of —-^M are purely imaginary and there is no damping 
(Fig. 19.2) 

c 

a k = i— sin&Z\x (19.44) 

Ax 

c 

uj k = — ia k = — sin kAx. (19.45) 

Ax 


19.2.2 Explicit Methods 


Time integration with an explicit forward Euler step proceeds according to (p. 293) 
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and can be formulated in matrix notation as 

f (t + At) = A f(f) = (1 - aM)f(t) (19.47) 

where the matrix M depends on the discretization method. 

19.2.2.1 Forward in Time, Backward in Space 

Combination with the backward difference quotient gives the FTBS (forward in time 
backward in space) method 

f{x, t + At) = f(x, t) — a if (x, t ) - fix - Ax, t)) (19.48) 

with the so called Courant number [243] 1 
At 

a = c -. (19.49) 

Ax 

The eigenvalues of 1 — aM are 
<7k = 1 - Q(1 - z lkAx ) 

= 1 — a(l — cos k Ax) + iasmkAx (19.50) 

with absolute square (Fig. 19.3) 

\a k \ 2 = 1 + 2(or — a)(l — cos k Ax). (19.51) 

Stability requires that \cr k \ < 1, i.e. 

2 ia 2 - a)(l - cos kAx) < 0 (19.52) 

and, since (1 — cos kAx) > 0 

ia — l)a < 0 (19.53) 

with the solution 2 

0 < a < 1. (19.54) 


'Also known as CFL (after the names of Courant, Friedrichs, Lewy). 

2 The so called Courant- Friedrichs- Lewy condition (or CFL condition). 
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Fig. 19.3 (Stability of the 
FTBS method) Top the 
magnitude of the eigenvalue 
\oic \ is shown as a function of 
k for positive values of the 
Courant number (from 
Bottom to Top) a = 

0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 

1.1. The method is stable for 
a < 1 (full curves) and 
unstable for a > 1 (dashed 
curves). Bottom the 
magnitude of the eigenvalue 
\ak \ is shown as a function of 
k for negative values of the 
Courant number (from 
Bottom to Top) a = 

-0.1, -0.3, -0.5, -0.7, 
—0.9. The method is 
unstable for all a < 0 


Fig. 19.4 (Performance of 
the FTBS method) An 
initially rectangular pulse 
(dashed cun’e) is propagated 
with the FTBS method 
(Ax = 0.01, At = 

0.005, a = 0.5). Due to 
numerical diffusion the 
shape is rapidly smoothened. 
Results are shown after 1,10 
and 100 round trips (2000 
time steps each) 


1.5 



0.5 


0 * .. 

-1 -0.5 0 0.5 1 



-1 -0.5 0 0.5 1 

k Ax/7t 



The FTBS method works, but shows strong damping due to numerical diffusion 
(Fig. 19.4). Its dispersion relation is 

u>kAt = —iln (ctk) = — iln ([1 — a(l — cos kAx) + ia sinkzLv]). (19.55) 

19.2.2.2 Forward in Time, Forward in Space 

For a forward difference we obtain similarly 


f(x, t + At) = f{x , t)-a { f(x + Ax, t) - fix, t)) 


(19.56) 
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a k = l - a (e~ ikAx - l) (19.57) 

\a k \ 2 = 1 + 2 (a 2 + a)(l - cos kAx) (19.58) 

which is the same result as for the backward difference with a replaced by—ct. 

19.2.2.3 Forward in Time, Centered in Space 

For a symmetric difference quotient, the eigenvalues of M are purely imaginary, all 
eigenvalues of (1 + aM) 

a k = 1 + iasxnkAx (19.59) 

\a k \ 2 = 1 + a 2 sin 2 kAx (19.60) 

have absolute values Icr* | > 1 and this method (FTCS, forward in time centered in 
space) is unstable (Fig. 19.5). 


19.2.2.4 Lax-Friedrichs-Scheme 

Stability can be achieved by a modification which is known as Lax-Friedrichs- 
scheme. The value of f (x, t ) is averaged over neighboring grid points 


fix , t + At) = 

1 — a 
-exp 


f (x + Ax) + fix — Ax) a 

— -- - - 2 (/(x + 4x) - f^ x ~ Ax)) 


( d\ 1 +a ( d\ 

\ Ax w x ) + ^r^\ X dx) 


fix , t) 


df iAx) 2 d 2 f 
a dx + 2 dx 2 


+ 


(19.61) 


Fig. 19.5 (Instability of the 
FTCS method) An initially 
Gaussian pulse (dashed 
cur\’e) is propagated with the 
FTCS method (Ax = 

0.01, At = 0.005, a = 0.5). 
Numerical instabilities 
already show up after 310 
time steps and blow up 
rapidly afterwards 



x/Ax 
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The error order is O (Ax 2 ) as for the FTCS method but the leading error has now 
diffusive character. We obtain the system of equations 


f (* + At) = - 


The eigenvalues follow from 


/ 1- 

- a 

1 + a\ 

1 + a 

1 — a 



1 + a 

1 — a 

y 1 — a 


1 + a ) 


f(0. 


(19.62) 


(l-a)e- i( " +mA,r + (l+a)e- 


and are given by 


[(1 - a)e 


+ (1 + a)e i<:/ljr ] 


(19.63) 


1 


(19.64) 


a k = - [(1 - a)e~ lkAx + (1 + a)j kAx ] 

= cos k Ax + ia sin kAx. 

The absolute square is 

\a k \ 2 = 1 [(1 - a)c-' )kAx + (1 + a)e ikAx ] [(1 - a)e i)kAx + (1 + a)e~ ikAx ] 


= ^ [(1 - a) 2 + (1 + a) 2 + (1 - a 2 )(e~ 2ikAx + e 2ikAxx 

= -[l + a 2 +(l — a 2 ) cos 2kAx\ 

= 1 — (1 — a 2 )(sin A'Ar) 2 

and the method is stable for 
(1 — a 2 )(sin kAx) 2 > 0 
which is the case if the Courant condition holds 


(19.65) 


(19.66) 


l«l < 1. 


(19.67) 
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19.2.2.5 Lax-Wendroff Method 


The Lax-Friedrichs method can be further improved to reduce numerical diffusion 
and obtain a method which is higher order in time. It is often used for hyperbolic 
partial differential equations. From the time derivative of the advection equation 


(9f\ = _ = 

\dt) C dx\dt) 


d_(df 
dt 


dx 2 


(19.68) 


we obtain the Taylor expansion 

df (At) 2 ? d 2 f 

f (t + At) = f (t) — At 4- -—c + ''' 

which we discretize to obtain the Lax-Wendroff scheme 

/(x + Ax, t) — f(x — Ax, t) 


f(x, t + At) = f(x, t) — At c- 


2 Ax 


(At) 2 2 f(x + Ax, t) + /(x — Ax, t) — 2 f(x, t) 
2 C (Ax) 2 ' 


(19.69) 


(19.70) 


This algorithm can also be formulated as a predictor-corrector method. First we 
calculate intermediate values at t + At/2, x ± Ax/2 with the Lax method 


Ax At 

f(x + —,t + —)■■ 

Ax At 

/ (.v--,r + -> 


f(x + Ax,t) + f(x,t) f(x + Ax,t)-f(x,t) 

- cAt - 

2 2 Ax 

f(x, t) - f(x - Ax, t) 


f(x, t) + f(x - Ax, t) 


cAt 


2 Ax 


(19.71) 


which are then combined in a corrector step 


f(x, t + At) — f(x, t) — cAt 


/(x + f,t + f)-/(x-f,r + f) 
Ax 


(19.72) 


Insertion of the predictor step (19.71) shows the equivalence with (19.70). 


cAt 

f(x,t) - — 
Ax 


f(x + Ax, t) + f(x, t) f(x + Ax, t) - f(x, t) 

- cAt - 


2 Ax 


f(x,t) + f(x-Ax,t) f(x,t)-f(x-Ax,t) 

-h cAt - 

2 2Z\x 
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= fix, t)-c 


At 
2 Ax 


f{x + Ax, t) — f{x — Ax, t) 


c 2 iAt) 2 

+ 2 (A x y- + A *’ ^ _ 2 /( x > t '> + fi x ~ Ax > 0 ] • 


In matrix notation the Lax-Wendroff method reads 


f (f + At) = 


a+ar | _,2 or—a. 

2 1 U 2 


m 


with eigenvalues 


J OC “j“ Ot jf. A (X OL j bAy 

a k = 1 - a 2 H-e 1 ^ H-e lkAx 

2 2 

= 1 — a 2 + a 2 cos kAx + ia sin kAx 


\a k \ 2 = (1 + a 2 icosikAx) — l)) 2 + a 2 sin 2 kAx 

= 1 - a 2 (l - ct 2 )(l - cos kAx) 2 
which is < 1 for 

a 2 (l — cr)(l — coskAx) 2 > 0 
which reduces to the CFL condition 


M < l. 

19.2.2.6 Leapfrog Method 

The Leapfrog method uses symmetric differences for both derivatives 

fix, t + At) — fix, t — At) fix + Ax, t) — fix — Ax, t) 

2At 2 Ax 


(19.73) 


(19.74) 


(19.75) 


(19.76) 


(19.77) 


(19.78) 


(19.79) 
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Fig. 19.6 Leapfrog method 
for advection 



to obtain a second order two step method 

fix, t + At) = f(x, t — At) — a [fix + Ax, t) — fix — Ax, t )] (19.80) 


on a grid which is equally spaced in space and time. 

The calculated data form two independent subgrids (Fig. 19.6). For long integra¬ 
tion times this can lead to problems if the results on the subgrids become different 
due to numerical errors. Introduction of a diffusive coupling term can help to avoid 
such difficulties. 

To analyze stability, we write the two step method as a one step method, treating 
the values at even and odd time steps as independent variables 


g” n = fim Ax, 2nAt) h" m — ffnAx, (2 n + l)At) 
for which the Leapfrog scheme becomes 
hi = h"- 1 ~ aig’! n+l - g n m _ x ) 


(19.81) 


(19.82) 


ft" +1 = 9 n m - <*ih n m+x - /C-t)- 

Combining this two equations we obtain the one step iteration 


(19.83) 


rln i„n r. 
9m J m Jr, 


2n+\ 


( hn \-( 1 -aM\(h"- l \ _ ( 1 —aM \ (li n ~ l \ 

\9 n+l ) ~ \-aM l) V 1 / V g n ) ~\-aM \+a 2 M 2 )\ g n ) 


'h n ~ v 
9 " 

(19.84) 


The eigenvalues of the matrix M are A* = —2i sin kAx, hence the eigenvalues of 
the Leapfrog scheme 
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Fig. 19.7 (Dispersion of first order explicit methods) Real (Left) and imaginary (Right) part of 
cj* are shown for the first order explicit FTBS (Top), FTCS (Middle) and Lax-Friedrich (Bottom) 
methods for values of a = 0.3, 0.5, 0.75, 0.9 


2x2 


&k — 1 + 


or A 


±Ja 2 A 2 + 


cr 4 A 4 


= 1 — 2cr sin" kAx ± 2 


.yja 2 sin 2 kAx(a 2 sin 2 kAx — 1). 


( 19 . 85 ) 


For | a | < 1 the squareroot is purely imaginary and 
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Fig. 19.8 (Dispersion of second order explicit methods) Real (Left) and imaginary (Right) part of 
04 - are shown for the second order explicit Lax-Wendroff (Top) and leapfrog ( Bottom) methods for 
values of a = 0.3, 0.5, 0.75, 0.9 


l ^| 2 = 1 


i.e. the method is unconditionally stable and diffusionless. The dispersion 

2\!or sin 2 kAx — a 4 sin 4 kAx 


2to At = — i In = arctanl ± 


(■ 


1 — 2a 2 sin 2 kAx 
has two branches. Expanding for small kAx we find 
UJ ±ck + • • • . 




(19.86) 


(19.87) 


Only the plus sign corresponds to a physical mode. The negative sign corresponds 
to the so called computational mode which can lead to artificial rapid oscillations. 
These can be removed by special filter algorithms [244, 245]. 

Figures. 19.7, 19.8 and 19.9 show a comparison of different explicit methods. 
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Fig. 19.9 (Comparison of 
explicit methods) The results 
from the FTBS, 
Lax-Friedrichs(L, green), 
Lax-Wendroff (LW, black) 
and leapfrog (LF, red) 
methods after 10 roundtrips 
are shown. Initial values 
(black dashed curves) are 
Gaussian (Top), triangular 
(Middle) and rectangular 
(Bottom). Ax = 0.01, 

At = 0.005, a = 0.5 




19.2.3 Implicit Methods 

Time integration by implicit methods improves the stability but can be time con¬ 
suming since inversion of a matrix is involved. A simple Euler backward step (13.4) 
takes the derivative at t + At 

f (t + At) = f(t) - aMf(t + At) (19.88) 


which can be formally written as 
f(t + At) = (l + QM)-'f(l). 


(19.89) 
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The Crank-Nicolson method (13.5) takes the average of implicit and explicit Euler 
step 

f (t + At) = i(t) - [f(t + At) + f(t)] (19.90) 

f (t + At) = (l + |m)~' (l - Hi). (19.91) 

Both methods require to solve a linear system of equations. 


19.2.3.1 First Order Implicit Method 


Combining the back steps in time and space we obtain the BTBS (backward in time, 
backward in space) method 


f (t + At) — 


/ 1 

-1 1 


1 + a 


-1 w 1 


1 1 


-1 1 


m. 


(19.92) 


The tridiagonal structure of the matrix 1 + aM simplifies the solution of the 
system. The eigenvalues of (1 + aM)~ x are 


(Tk 


1 

1 + a(l - e kAx ) 


(19.93) 


Wk\ 2 = 


1 


(1 + a) 2 + a 2 — 2a(l + a) cos (kAx) 

and the method is unconditionally stable. 


< 1 


(19.94) 


19.2.3.2 Second Order Crank-Nicolson Method 

The Crank-Nicolson method with the symmetric difference quotient gives a second 
order method 


(l + “m) f(r + At) = (l - “m) f(t) 


(19.95) 
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The eigenvalues of (1 + | M) 1 (1 — f M) are 


1 + f i sinA:Z\x 


(19.96) 


1 — f i sink Ax 


1 — '-j- sin 2 kAx + i a sin kAx 


(19.97) 


1 + Sp- sin 2 kAx 


with 


Kl 2 = l. 


(19.98) 


There is no damping but strong dispersion at larger values of a, slowing down 
partial waves with higher k —values (Fig. 19.1 1) 

This method is unconditionally stable (Fig. 19.10). It may, however, show oscilla¬ 
tions if the time step is chosen too large (Fig. 19.1 1). It can be turned into an explicit 
method by an iterative approach (iterated Crank-Nicolson method, see p. 475), which 
avoids solution of a linear system but is only stable for a < 1. 


19.2.4 Finite Volume Methods 

Finite volume methods [246] are very popular for equations in the form of a 
conservation law 



(19.99) 



o 


0.5 


0 3 , 

£ 


- 0.5 


-1 


0 0.2 0.4 0.6 0.8 

kAx/7t 


0 0.2 0.4 0.6 0.8 

kAx/n 


Fig. 19.10 (Dispersion of the Crank-Nicolson method) Real (Left) and imaginary part (Right) 
part of u>k are shown for a = 0.1, 9, 5. 10. This implicit method is stable for a > 1 but dispersion 
becomes noticeable at higher values 










446 


19 Advection 


Fig. 19.11 (Performance of 
the Crank-Nicolson method) 
Results of the implicit 
Crank-Nicolson method 
after 10 roundtrips are 
shown. Initial values ( black 
dashed cun’es) are Gaussian 
(Top), triangular (Middle) 
and rectangular (Bottom). 

Ax = 0.01, At = 0.01 
(a = 1, red dash-dotted 
curve) At = 0.1(a = 10, 
blue full carve) 

At = 0.2 (a = 20, green 
dotted curve , only shown for 
Gaussian initial values) 



In one dimension the control volumes are intervals, in the simplest case centered at 
equidistant grid points x n 


Ax Ax 

Vn = I Art ^ 1 ^11 “h ^ ] * 


(19.100) 


Integration over one interval gives an equation for the cell average 


dfn(0 

dt 


1 d r x "+ Ax / 2 


. „ , f{x,t) = - — 

Ax dt J Xn -Ax/2 Ax . 


1 nx„+Ax/2 Q 


— /( a -, t) 

'x n -Ax/2 OX 
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Fig. 19.12 (Finite volume 
method) The change of the 
cell average f n is balanced 
by the fluxes through the cell 
interfaces J„± y 2 


J n-1/2^) J n+1/2W 



X 


n-1 


X 


1 

Ax 


/\y /\ X 

J(x„ + —,t)-J(x„-—,t) 


(19.101) 


Formally this can be integrated 


— — i r r ,+Al / \ r ,+Al / ax \ i 

f n (t + At)-f(t) = ~— j + 2 , t'jdi - j' j[x n - —,Adt\^ 

(19.102) 


and with the temporally averaged fluxes through the control volume boundaries 

(19.103) 


1 rl+At 

Jn±1/l(t) = ~At J t 


( \ 

J ( x n ± t J dt 


it takes the simple form (Fig. 19.12) 


f„(t + At) = f„(t) - — [y„+i/ 2 (f) - J n -y 2 (t )] . (19.104) 

A numerically scheme for a conservation law is called conservative if it can 
be written in this form with some approximation J„±i/ 2 (t) of the fluxes at the cell 
interfaces. Conservative schemes are known to converge to a weak solution of the 
conservation law under certain conditions (stability and consistency). 

To obtain a practical scheme, we have to approximate the fluxes in terms of the cell 
averages. Godunov’s famous method [247] uses a piecewise constant approximation 
of fix, t) 

f{x, t) ^ f n (t) for .r„_i/ 2 < x < x„+i/ 2 . (19.105) 

To construct the fluxes, we have to solve the Riemann problem' 

df _ dJ 
dt dx 


3 A conservation law with discontinuous initial values. 


(19.106) 
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with discontinuous initial values 


f(x, t) 


Jn(t) if X < X n+ l/2 
fn+ l(0 lfx > X n +'/2 


in the time interval 


t < t' < t + At. 


(19.107) 


(19.108) 


For the linear advection equation with J(x, t ) = cf(x, t) the solution is eas¬ 
ily found as the discontinuity just moves with constant velocity. For cAt < Ax, 
d’Alembert’s method gives 

f(x n + ^-,t’)=J n (t) (19.109) 

and the averaged flux is 


Jn+]/l(t) ~ At J, 


t+At Ax — 

f(Xn + —, t')dt = Cf n {t). 


(19.110) 


Finally we end up with the FTBS upwind scheme (Sect. 19.2.2) 
7 n(t + = J n (t) - ^ [fjt) - /„_!(/)] • 


(19.111) 


For general conservation laws, approximate methods have to be used to solve the 
Riemann problem (so called Riemann solvers [248]). 

Higher order methods can be obtained by using higher order piecewise interpo¬ 
lation functions. If we interpolate linearly 

/CM) ^ fn(t) + (x ~ x n )a n (t) for x„_i /2 < x < x n+i/2 

the solution to the Riemann problem is 


f(x n = f(x„ + ^~ - c(t' - f), t) = f n (t) + 


Ax , 

- c(t — t) 

2 v 


o~n (0 


= fn(t) + 


Ax 

- c(t' - r) 

2 


fn+l ~ fn 


2 Ax 
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The time averaged fluxes are 
Jn+ 1/2 = cf n {t) + C 
and we end up with 



— — cAt 

f n (t + At) = f n (t)- — 


( Ax At\ 

— -c—j (o-„-cr„_i) 


(19.112) 


If we take the slopes from the forward differences 

7„+l - fn 

&n — . 

Ax 

we end up with 


(19.113) 


— — cAt r — — cAt f„ ,i —2 f„ + i 

fn(t + At) = /„(*) - — [, f n (t) - /„_!(;)] - — (Ax- cAt ) ^ ^ J "-' 


— cAt 

= fn(t) - -r~ 

Ax 


fn(0 — fn- 1(0 + 


fn +1 _ 2 /„ + /„_! 


= /b(0 - 


cAf 

Ax 7 


T/j+1 f n — \ 

2 


(cAQ 2 /„ +1 — 2/„ +/„_i 
2Aj Ax 


(cAQ 2 /„ +1 - 2/„ + /„_! 
2Ax Ax 

(19.114) 


i.e. we end up with the Lax-Wendroff scheme. Different approximations for the slopes 
are possible (backward difference, symmetric differences) leading to the schemes of 
Fromm and Beam-Warming. 


19.2.5 Taylor-Galerkin Methods 


The error order of finite difference methods can be improved by using a finite element 
discretization [249, 250]. We start from the Taylor series expansion in the time step 


f (t + At) — f (t) + At 


9f 

dt 


(At) 2 d 2 f 

2 dt 2 


(At) 2 d 3 f 
6 dt 3 


(19.115) 


which is also the basis of the Lax-Wendroff method (19.70) and make use of the 
advection equation to substitute time derivatives by spatial derivatives 
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f{t + At) = fit) - At M + 

OX 


jAt) 2 2 d 2 f (At ) 3 2 d 3 f 

2 dx 2 6 dtdx 2 


(19.116) 


where we use a mixed expression for the third derivative to allow the usage of linear 
finite elements. We approximate the third derivative as 


d 3 / = d 2 fjx,t + At)-fjx,t) 
dtdx 2 dx 2 At 


(19.117) 


and obtain an implicit expression which is a third order accurate extension of the 
Lax-Wendroff scheme 


iAt) 


2 02 ' 
_ C 2 — 

6 dx 2 


ifix, t + At) 


fix, t)) = -Atc^f- 
dx 


iAt) 2 2 d 2 f 


dx 2 ' 

(19.118) 


Application of piecewise linear elements on a regular grid (p. 282) produces the 
following Lax-Wendroff Taylor-Galerkin scheme 


1 + - (1 - a 2 ) D 2 


(f it + At) - fit)) = 


-clM\ + ^ M 2 


fit). (19.119) 


The Taylor-Galerkin method can be also combined with other schemes like 
leapfrog or Crank-Nicolson [250]. It can be generalized to advection-diffusion prob¬ 
lems and it can be turned into an explicit scheme [251] by series expansion of the 
inverse in 


f it + At) — f(f) + 


io 


2 )m 2 


- 1-1 


-Q. M\ + M 2 


m. 


(19.120) 


The eigenvalues are 


&k — 1 + 


ai sin A: Ax — 2a 2 sin 2 ^4^ 
1 — |(1 — ct 2 ) sin 2 


(19.121) 


The method is stable for \a\ < 1. Due to its higher error order it shows less 
dispersion and damping than the Lax-Wendroff method (Fig. 19.13) and provides 
superior results (Fig. 19.14). 
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_i_i___ 1 — —i— z — 

0 0.2 0.4 0.6 0.8 1 

kAx/jt 


Fig. 19.13 (Dispersion of the Taylor-Galerkin Lax-Wendroff method) Real ( Left ) and imaginary 
part (Right) part of ui^ are shown for a = 0.3, 0.5, 0.75, 0.9 


Fig. 19.14 (Performance of 
the Taylor-Galerkin 
Lax-Wendroff method) 
Results of the Lax-Wendroff 
(dashed cun’es) and 
Taylor-Galerkin 
Lax-Wendroff (full curves) 
methods are compared after 
25 roundtrips (2000 steps 
each). Ax = 0.01, 

At = 0.005, a = 0.5 



19.3 Advection in More Dimensions 


While in one dimension for an incompressible fluid c = const, this is not necessarily 
the case in more dimensions. 
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19.3.1 Lax-Wendroff Type Methods 

In more dimensions we substitute 

^ = -u V/ (19.122) 

at 

w = I (?) = - v (I) = (UV) "' V,/ <19 - 123) 

in the series expansion 


df At 2 d 2 f 

fit + At) - /(f) = At-± + — + ■ ■ ■ (19.124) 

to obtain a generalization of the Taylor expansion (19.69) 


f{t + At) - fit) = -AtuVf + —(uV)(uV)/ + ■ • • (19.125) 

which then has to be discretized in space by the usual methods of finite differences 
or finite elements [250]. Other one-dimensional schemes like leapfrog also can be 
generalized to more dimensions. 


19.3.2 Finite Volume Methods 

In multidimensions we introduce a, not necessarily regular, mesh of control vol¬ 
umes Vj. The surface of V, is divided into interfaces ,4, /f to the neighboring cells. 
Application of the integral form of the continuity equation (19.3) gives 


d_ 

dt 


dV fir, t) — — ® J(r, t)d A 

'dVi 


and after time integration 


fi it + Af) - fit) = - At y' Ji,ait) 

a 


(19.126) 


(19.127) 


with the cell averages 
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Fig. 19.15 Averaged fluxes 
in 2 dimensions 




J i,1 









J 

i,3 



^ [ dv f(r,t) (19.128) 

Vi at J v . 

and the flux averages 

J j j-t+At r 

//,a(0= —-X/ dt ' f A J^ f VA. (19.129) 

For a regular mesh with cubic control volumes the sum is over all cell faces 

fijktt + At) = f ijk(t) — At \ Ji+l/ 2 ,j,k(t) + Jij+l/2,k(t) + Ji,j,k+ 1/2 ,(0 

-li-yuAt) - - Jkj.k-'/iit)] ■ (19.130) 

The function values have to be reconstructed from the cell averages, e.g. piecewise 
constant 


/(r, t) = fj(t) for reV,- (19.131) 

and the fluxes through the cell surface approximated in a suitable way, e.g. constant 
over a surface element (Fig. 19.15) 


J(r, t) = Ji, a (t) for r e A ijQ ,. (19.132) 

Then the Riemann problem has to be solved approximately to obtain the fluxes for 
times t... t + At. This method is also known as reconstruct evolve average (REA) 
method. An overview of average flux methods is presented in [252]. 
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19.3.3 Dimensional Splitting 

Splitting methods are very useful to divide a complicated problem into simpler 
steps. The time evolution of the advection equation can be written as a sum of three 
contributions 4 


df d(u x f ) d(u y f) d{u z f) 

— = -d.v(u/) = — - - - — 


(19.133) 


or, for an incompressible fluid 


df df df df 

— = -u grad f = -u x - - u y - - u z — 

dt dx ' dy dz 


(19.134) 


which has the form 


% = Af = (A x + A y + A z )f. 


(19.135) 


The time evolution can be approximated by 


f(t + At) = e A,A f(t) 


-At A xa At A ya At A. 


fit ) 


(19.136) 


i.e. by a sequence of one-dimensional time evolutions. Accuracy can be improved 
by applying a symmetrical Strang-splitting 


fit + At) « e At/2A Ie At/2Ay e AtA Ze At/2Ay e At/2A x j?( t y 


(19.137) 


Problems 

Problem 19.1 Advection in one Dimension 

In this computer experiment we simulate 1-dimensional advection with periodic 
boundary conditions. Different initial values (rectangular, triangular or Gaussian 
pulses of different widths) and methods (Forward in Time Backward in Space, 
Lax-Friedrichs, leapfrog, Lax-Wendroff, implicit Crank-Nicolson, Taylor-Galerkin 
Lax-Wendroff) can be compared. See also Figs. 19.4, 19.11, 19.14 and 19.9. 


4 This is also the case if a diffusion term D 



a 2 / 

W 


a 2 .A 

~d?) 


is included. 






Chapter 20 

Waves 


Waves are oscillations that move in space and time and are able to transport energy 
from one point to another. Quantum mechanical wavefunctions are discussed in 
Chap. 23. In this chapter we simulate classical waves which are, for instance, impor¬ 
tant in acoustics and electrodynamics. We use the method of finite differences to 
discretize the wave equation in one spatial dimension 


drf 


f(t,x) = c 2 


d 2 
d.rf 


f(t,x). 


( 20 . 1 ) 


Numerical solutions are obtained by an eigenvector expansion using trigonomet¬ 
ric functions or by time integration. Accuracy and stability of different methods are 
compared. The wave function is second order in time and can be integrated directly 
with a two step method. Alternatively, it can be converted into a first order system 
of equations of double dimension. Here, the velocity appears explicitly and veloc¬ 
ity dependent damping can be taken into account. Finally, the second order wave 
equation can be replaced by two coupled first order equations for two variables (like 
velocity and density in case of acoustic waves), which can be solved by quite general 
methods. We compare the leapfrog, Lax-Wendroff and Crank-Nicolson methods. 
Only the Crank-Nicolson method is stable for Courant numbers a > 1. It is an 
implicit method and can be solved iteratively. In a series of computer experiments 
we simulate waves on a string. We study reflection at an open or fixed boundary and 
at the interface between two different media. We compare dispersion and damping 
for different methods. 


20.1 Classical Waves 

In classical physics there are two main types of waves: 

Electromagnetic waves do not require a medium. They are oscillations of the elec¬ 
tromagnetic field and propagate also in vacuum. As an example consider a plane wave 
which propagates in x-direction and is linearly polarized (Fig. 20. 1). The electric and 
magnetic field have the form 
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Fig. 20.1 Electromagnetic 
wave 


E = [ E y (x, t) 


B 



Maxwell’s equations read in the absence of charges and currents 

<m 0E 

divE = div B = 0, rot E = ——, rotB = uoSo--. 

dt dt 


(20.3) 


The fields (20.2) have zero divergence and satisfy the first two equations. Application 
of the third and fourth equation gives 


dE y _ dB- dB z _ dE y 
dx dt dx ^° ° dt 


which can be combined to a one-dimensional wave-equation 

d 2 E Y 


dt 2 


y _ 2 d 2 E y 
dx 2 


(20.4) 


(20.5) 


with velocity c = (/io£o) ^ 2 - 

Mechanical waves propagate through an elastic medium like air, water or an elastic 
solid. The material is subject to external forces deforming it and elastic forces which 
try to restore the deformation. As a result the atoms or molecules move around their 
equilibrium positions. As an example consider one-dimensional acoustic waves in 
an organ pipe (Fig. 20.2): 

A mass element 


dm = gdV = gAdx (20.6) 

at position x experiences an external force due to the air pressure which, according to 
Newton’s law changes the velocity v of the element as described by Euler’s equation 1 


1 we consider only small deviations from the equilibrium values £>o- po , vq = 0. 
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F(x)=p(x)A 


F(x+dx)=-p(x+dx)A 


dm 


J(x)= p 0 Av(x) 

J(x+dx)= p 0 Av(x+dx) 


x x+dx 


Fig.20.2 (Acoustic waves in one dimension) A mass element dm = gAdx at position x experiences 
a total force F = F(x) + F{x + dx) = —A^dx. Due to the conservation of mass the change of 
the density ^ is given by the net flux J = J(x) — J(x + dx) = —ggA^dx 


d dp 

00 dt V ~ dx ' 

The pressure is a function of the density 

P_^(S_ Y' = n El =c 2 

Po \QoJ \dgJo go 


(20.7) 


( 20 . 8 ) 


where n = 1 for an isothermal ideal gas and n ~ 1.4 for air under adiabatic conditions 
(no heat exchange), therefore 


eo Wt v = 


dg 
dx' 


(20.9) 


From the conservation of mass the continuity equation (12.10) follows 


Ft g = - go d- x v - 


( 20 . 10 ) 


Combining Ihe time derivative of (20.10) and the spatial derivative of (20.9) we 
obtain again the one-dimensional wave equation 


d 2 2 d 2 
di 20 ~d^ 0 ' 

The wave-equation can be factorized as 


( 20 . 11 ) 


Id d\(d 
\di + C ]h) V* 


d\ (d 

C dx) 9 ~ \dt 



( 20 . 12 ) 


which shows that solutions of the advection equation 


( 


d 

di 


d \ 

±C d^J 


g — 0 


(20.13) 
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Fig. 20.3 d’Alembert 
solution to the wave equation 


t 



are also solutions of the wave equation, which have the form 

g=f(x±ct). (20.14) 

In fact a general solution of the wave equation is given according to d’Alembert as 
the sum of two waves running to the left and right side with velocity c and a constant 
envelope (Fig. 20.3) 

6 = f\ (x + ct) + f 2 (x - ct). (20.15) 

A special solution of this kind is the plane wave solution 


f(x,t) = e' ut±ikx 


with the dispersion relation 
lo = ck. 


(20.16) 


20.2 Spatial Discretization in One Dimension 


We use the simplest finite difference expression for the spatial derivative (Sects. 3.4 
and 12.2) 


d 2 f(t,x + Ax) + f(t,x-Ax)-2f(t,x) 2 

a? f{x - ,) = - a ?- + 0(Ax) 


(20.17) 


and a regular grid 


x m = mAx m — 1,2... M 


(20.18) 


fm — f ( Xm ) ■ 


(20.19) 
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(a) 




(c) 



x 0 X 1 x 2 


X M-1 X M X M+1 



Fig. 20.4 (Boundary Conditions for 1 -dimensional waves) Additional boundary points 
xq.-Vm+i are used to realize the boundary conditions, (a) Fixed boundaries /(xq) = 0 


r(/fe) - 2/(xi)) or/ (xm+i) = 0 ,^/(.t M ) = 


2/(xM)) (b) Periodic boundary conditionsxo = xm-. 


2/(xi))x m +i = xi,£j/(x m ) = 


aries^/Xx!) = 


dx 2 

_ f(x 2 )-f(x 0 ) _ 

2Ax 


= o^ /( x M ) = 

aries/(x 0 , f) = fo(f)#r/(xi) = 


1) - 

^/(Xi) = ^(/(X 2 ) + /(xm) - 
(/(*M-t) + /(*t) - 2 /(x wm ))-(c) Open bound- 
^t(2/(x 2 ) - 2/(xi)) or |/(x«) = 
(2/ ((xm-i) - 2/ (xm)). (d) Moving bound- 
(/fe)-2/(x 1 ) + ^ 0 (0)or /(xm+i,0 = £m+i(0. 


At 2 

0 ^/(xi) = 


1 

Ax 3 




(/(*m-i) - 2f(x M )+£ N+ i(t)) 


This turns the wave equation into the system of ordinary differential equations 
(Sect. 12.2.3) 


tl j. 2 fm+l T fm — 1 2 fff: 

At 2 m Ax 2 


( 20 . 20 ) 


where /o and /m+i have to be specified by suitable boundary conditions (Fig. 20.4). 
In matrix notation we have 


f(0 = 



(20.21) 
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d t 2 


f (r) = Am + s (n 


where for 


( 20 . 22 ) 


(-2 1 
1 -2 1 
1 -2 1 


fixed boundaries A = 


1 -2 1 
1 - 2 ) 


Ax 2 


S(0 = 0 (20.23) 


periodic boundaries - A = 


1-2 1 

1 -2 1 

1 -2 1 


1 \ 


V i 


1 -2 1 

1 - 2 ) 


Ax 2 


S (t) = 0 (20.24) 


1—2 2 
1-2 1 
1 -2 1 


open boundaries A = 


1 -2 1 
2 - 2 / 


^ S(?) = 0 (20.25) 



1-2 1 

1 -2 1 

1 -2 1 

\ 

c 2 

/ £ o (0 \ 
0 

moving boundaries A = 

1 

-2 1 

1 - 2 ) 

A? SW = 

0 

V£m+i(0/ 


(20.26) 


A combination of different boundary conditions for both sides is possible. 

Equation (20.20) corresponds to a series of mass points which are connected by 
harmonic springs (Fig. 20.5), a model, which is used in solid state physics to describe 
longitudinal acoustic waves [253]. 


2 This corresponds to the boundary condition /o = / 2 . Jy fix ]) = 0. Alternatively we could use 
/o = f\ , i/ 2 ) = 0 which replaces the 2 s in the first and last row by 1 s. 
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'f 













X 


(j-l)Ax jAx (j+1)Ax 


Fig. 20.5 (Atomistic model for longitudinal waves) A set of mass points m is connected by springs 
with stiffness K. The elongation of mass point number j from its equilibrium position xj = jAx 
is The equations of motion = —K(£j — £;_i) — AT(£; — £;+i) coincide with (20.20) with 

a velocity of c = AxJ^Ax 


20.3 Solution by an Eigenvector Expansion 


For fixed boundaries (20.20) reads in matrix form 

^f(0 = Am 

with the vector of function values: 

/ Mt)\ 

m = ; 

\ 1m (t ) j 

and the matrix 


(20.27) 


(20.28) 


1-2 1 

1 -2 1 

1 -2 1 


A = 


V 


1 -2 1 

1 - 2 / 


c 

Ax 2 


(20.29) 


which can be diagonalized exactly (Sect. 10.3). The two boundary points /(0) = 0 
and f{{M + 1) Ax) = 0 can be added without any changes. The eigenvalues are 


/~2 / k /\ x \ 

= 2 ——^ (cos (kAx) — 1) = — 7 sin 2 ( —— | = (i 


A = 2-=-(cos (kAx) — 1) 

Ax 2 


ixl 


, , ( iuii r kAx = 

V 2 J (M + l) 


/= 1...M 

(20.30) 


with the frequencies 
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Fig. 20.6 (Dispersion of the 
discrete wave equation) The 
dispersion of the discrete 
wave equation approximates 
the linear dispersion of the 
continuous wave equation 
only at small values of k. At 
k max = 7r/At it saturates at 
^max = 2c / Ax — 

(2/7r) ck max 



2 c 

to ,t = — sin 
Ax 



(20.31) 


This result deviates from the dispersion relation of the continuous wave equation 
(20.11) u>k = ck and approximates it only for kAx <<C 1 (Fig. 20.6). 

The general solution has the form (Sect. 12.2.4) 


fn(t) = X i C i+ e ‘' 


,UOl t 


1=1 


+ C/_e"' w ' 



nl \ 

(M + 1)/ 


(20.32) 


The initial amplitudes and velocities are 

M / I \ 

fn (t = 0) = y (C/+ + C/_) sin ^ J = F m 

= 0, x m ) = y]iui (C/+ - C/_) sin (m / ) = G m (20.33) 

df “ V (M+l)/ 

with and G m given. Different eigenfunctions of a tridiagonal matrix are mutually 
orthogonal 



irl 

M + 1 



ttI' \ 
M + l) 



(20.34) 


and the coefficients C/± follow from a discrete Fourier transformation: 
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1 ^ / 7 Tl \ 

Fi = — > sin I m - I F m 

1 M \ N + l I 

m =1 

1 / 7 rl' \ ( 7r/ \ 1 

= — > > (C/'+ + CV_) sm I m -) sin ( m -) = -(C/ + + C/_) 

M ■“ ■“ v + ' V M + 1 / V M + l / 2 /+ ' 

m=l/'=l v 7 v 7 

(20.35) 

— 1 / -kI \ 

Gi = — > sin I m - I G n 

1 M \ N + 1 / 

m=l 

i ^ / tt/' \ / 7r/ \ I 

= iu i ( c '+ - c '-> sin ( m M +i ) sin ( m ) = 2 i ' u;/(C/+ “ Q - } 

m=l/'=l v 7 v 7 

(20.36) 

Q+ = Fi + -—Gi 

liOl 

Ci- = Fj -— G/- (20.37) 

lUJl 

Finally the explicit solution of the wave equation is 

/m(0 = X! 2 ^' cos ( w ' f ) H-- sin (a;/ 0) sin ( m / ) . (20.38) 

rr <*>, V M + i/ 


Periodic or open boundaries can be treated similarly as the matrices can be diag¬ 
onalized exactly (Sect. 10.3). For moving boundaries the expansion coefficients are 
time dependent (Sect. 12.2.4). 


20.4 Discretization of Space and Time 


Using the finite difference expression also for the second time derivative the fully 
discretized wave equation is 

f(t + At, x) + f(t — At, x) — 2 f(t, x) 

At 2 


_ ^ f{t,x + Ax) + f(t,x - Ax) - 2f(t, x) | ^ 2 

Ax 2 


(20.39) 


For a plane wave 




/ = e 


(20.40) 


we find 
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Fig. 20.7 (Dispersion of the 
discrete wave equation) Only 
for a = 1 or for small values 
of kAx and uAt is the 
dispersion approximately 
linear. For a < 1 only 
frequencies 

u < u^ m ax = 2 arcsin(a)/2\/ 
are allowed whereas for 
a > 1 the range of k-values 
is bounded by 
A'max = 2arcsin(l/a)/2lx 



^iuAt g—i uAt 


At ^ 

2 = c 2 ^ (e ikAx 
Ax 2 v 


^—ikAx 


2 ) 


(20.41) 


which can be written as 


u>At kAx 

sin-= a sin- 

2 2 


(20.42) 


with the so-called Courant-number [243] 

At 

a — c -. (20.43) 

Ax 

From (20.42) we see that the dispersion relation is linear only for a — 1. For 
a / 1 not all values of co and k allowed (Fig. 20.7). 


20.5 Numerical Integration with a Two-Step Method 


We solve the discrete wave equation (20.39) with fixed or open boundaries for 

f(t + At, x) = 2f(t, x)(l — a 2 ) + or(/(f, x + Ax) + f(t, x — Ax)) 

- f{t - At, x) + 0(At 2 , Ax 2 ) (20.44) 


on the regular grids 

x m — mAx m = 1,2 ... M 


(20.45) 


t n — nAt n = 1,2 ... N 


(20.46) 
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tn = 


//f\ 


Kfu) 


i) 


(20.47) 


. f (f/i i Xm) 

by applying the iteration 

rn+1 OTI ^2\ rn , _ 2 rn , _ 2 rn rn— 1 

Jm —2(1 - a )f m + a / m+1 + a / w _i - J m • 


(20.48) 


This is a two-step method which can be rewritten as a one-step method of double 
dimension 


Or)- r (£M 2+ ^,')(£) 


(20.49) 


with the tridiagonal matrix 


/-2 a i 

1 -2 1 


M = 


V 


1 -2 1 
oat —2 f 


(20.50) 


where nqand fly have the values 1 for a fixed or 2 for an open end. 
The matrix M has eigenvalues (Sect. 10.3) 


A = 2 cos (kAx) — 2 = 


-4 sin 2 ^ 


kAx \ 
2 /' 


(20.51) 


To simulate excitation of waves by a moving boundary we add one grid point with 
given elongation £o (f) and change the first equation into 

/(?„+!, xi) = 2(l-a 2 )f(t n ,xi)+a 2 f(t n ,X2)+a 2 £o(tn)-f(t n -i,xi). (20.52) 

Repeated iteration gives the series of function values 


(£) ■ 00 - r (£) ■ 00-^00 ■ 


(20.53) 


A necessary condition for stability is that all eigenvalues of T have absolute values 
smaller than one. Otherwise small perturbations would be amplified. The eigenvalue 
equation for T is 


'2 + a 2 M — a — 




(20.54) 
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Fig. 20.8 (Stability regions 
of the two-step method) 
Instabilities appear for 
\a\ > 1. One of the two 
eigenvalues a becomes 
unstable (|cr| > 1) for waves 
with large k-values 



k Ax 


We substitute the solution of the second equation 

u = (tv (20.55) 

into the first equation and use the eigenvectors of M (Sect. 10.3) to obtain the eigen¬ 
value equation 

(2 + a 2 A - <7)0-0 -v=0. (20.56) 


Hence a is one of the two roots of 


cr 2 - cr (cr A + 2) + 1 = 0 
which are given by (Fig. 20.8) 



we find 


-4 < A 
1 -2a 2 


< 0 

a 


2 A 


< ~ 2 ^ 1 < 


1 


(20.57) 


(20.58) 


(20.59) 


(20.60) 
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and the square root in (20.58) is imaginary if 



(20.61) 


which is the case for 



(20.62) 


This holds for all k only if 


M < i. 

But then 


(20.63) 



(20.64) 


and the algorithm is (conditionally) stable. If on the other hand \a\ > 1 then for 
some k-values the square root is real. Here we have 



< -1 


(20.65) 


and finally 



( 20 . 66 ) 


which shows that instabilities are possible in this case. 


20.6 Reduction to a First Order Differential Equation 

A general method to reduce the order of an ordinary differential equation (or a 
system of such) introduces the time derivatives as additional variables (Chap. 13). 
The spatially discretized one-dimensional wave equation (20.22) can be transformed 
into a system of double dimension 



(20.67) 
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lv(0 = -^M f (r) + S(r). 

dr Ax 2 


We use the improved Euler method (Sect. 13.5) 
f(r + At) — f(r) + v(r -+——)At + O(Ap) 


( 20 . 68 ) 


(20.69) 


v(r + At) = \(t) + 


At 


At 


—Mf(r + T ) + S(r + T ) 


and two different time grids 

f„=f(r„) S„=S(r„) ii — 0, 1... 

f(Wt) = f(*n) + v(tn+i/l)At 

\„ = v(r„_i/ 2 ) n — 0, 1... 


v(r„+ 1 /2 ) = v(r„_i /2 ) + 


—Mf(r„) + S(r„) 


/4r. 


2\r + 0(z\r 3 ) (20.70) 

(20.71) 

(20.72) 

(20.73) 

(20.74) 


We obtain a leapfrog (Fig. 20.9) like algorithm (p. 398) 

A 


v„ +1 = v„ + 


Ax 2 

f«+i — fn “f w+t^ir 


Mf„ + S„ 


z\r 


(20.75) 

(20.76) 


where the updated velocity (20.75) has to be inserted into (20.76). This can be 
combined into the iteration 

/f„+i\ = /f,+v„4? + [^ I Mf„+S„]zir 2 \ = / 1 + l ^£m At\ / fn \ /S„Ar 2 \ 

I + ] \ ljVvJ \S„A, ) 


(20.77) 


Fig. 20.9 Leapfrog method 


(n-1/2)At (n+1/2) At 
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Since the velocity appears explicitly we can easily add a velocity dependent damp¬ 
ing like 

-7 v(t n ,x m ) (20.78) 

which we approximate by 

-7 v(t n -^-,x m ) (20.79) 

under the assumption of weak damping 

jAt <sC 1 . (20.80) 


To study the stability of this algorithm we consider the homogeneous problem 
with fixed boundaries. With the Courant number a = ^ (20.77) becomes 


( { n+ i\ = ( 1 + c ‘ 2m 4l(l-74()\/f,\ 

W»+t/ V 1-7 At 

Using the eigenvectors and eigenvalues of M (Sect. 10.3) 



(20.81) 


(20.82) 


we find the following equation for the eigenvalues a: 

(1 + a 2 X — a)u + At{l — jAt)v = 0 

a 2 Am + At( 1 — 7 At — cr)v = 0. (20.83) 

Solving the second equation for u and substituting into the first equation we have 


[(1 +a 2 A-<7) 


At 

—cr 2 A 


(1 — 7 At — a) + At( 1 — 7 At)] = 0 


(20.84) 


hence 


(1 + a 2 A — cr)(l — 7 At — a) — a 2 A(l — 7 At) = 0 
tx 2 — a (2 — 7 At + a 2 A) + (1 — 7 At) — 0 

A 1 


7 At a 2 A // 7 At a 2 A\“ 


(20.85) 


Instabilities are possible if the square root is real and cr < — 1. (cr > 1 is not possible). 
This is the case for 


7 At /- 7 At or\ / - 7 At 

A + ^-xt-Jl-jAt < 1-JL- + — <Jl- 1 At?*l-± Y - (20.86) 
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crA 

—2 + 7 At < —— < 0. 

The right inequality is satisfied, hence it remains 


a 2 sin 


in 2 ( 


kAx\ 

2 / 


< 1 


jAt 


(20.87) 


( 20 . 88 ) 


This holds for all k-values if it holds for the maximum of the sine-function 

a 2 < 1 — (20.89) 

This shows that inclusion of the damping term even favors instabilities. 


20.7 Two Variable Method 

For the 1 -dimensional wave equation (20.11) there exists another possibility to reduce 
the order of the time derivative by splitting it up into two first order equations similar 


to (20.9, 20.10) 


9= Qj 

* 

II 

c> 

81 os 

X 

(20.90) 

§?| cu 
5s 

X 

II 

C5 

§1 Q5 

'-S 

* 

(20.91) 


Several algorithms can be applied to solve these equations [254]. We discuss only 
methods which are second order in space and time and are rather general methods 
to solve partial differential equations. The boundary conditions need some special 
care. For closed boundaries with f (x o) = 0 obviously §f (jco) = 0 whereas §f(xo) 
is finite. Hence a closed boundary for f(t, x) is connected with an open boundary 
for g(t,x ) with gf(^o) = 0 and vice versa. This is well known from acoustics 
(Fig. 20.10). 



Fig. 20.10 (Standing waves in an organ pipe) At the closed (Left) end the amplitude of the longi¬ 
tudinal velocity is zero whereas the amplitudes of pressure and density changes are extremal. This 
is reversed at the open (Right) end 
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20.7.1 Leapfrog Scheme 


We use symmetric differences (Sect. 3.2) for the first derivatives 


fit + ^-,x)~ fit - 4r,x) git , * + 4r) - gix - 4r) 


At 


Ax 


2 +0(Ax 2 ,At 2 ) 


(20.92) 


git + x) - g(t - f,x) f(t,x + #) - /(x - 


At 


to obtain the following scheme 


Ax 


2 + 0(Ax 2 , At 2 ) 


(20.93) 


giitn+i/l. XmJf-lff) — git n -lj 2 , Xm+l/f) d“ OL if it n , X'm+l)) f it n , X m _l) 

f itn+1. X m ) == f (t n . X m ) -}- O; (t/(t;; + l /2' giitn+ l /2. Xtn—^/if) ■ 


Using different time grids for the two variables 


f„ = 


//f\ 


\/m/ 


/(U,xi) \ 


Jitn.Xu) / 


g» 


/ 9i \ 


giitn- 1 / 2 , Xl/ 2 ) \ 
. gitn-'fl. X M —1/2) / 


(20.94) 

(20.95) 


(20.96) 


this translates into the algorithm (Fig. 20.1 1) 

< +I = <+«(/,:; (20.97) 

/" +I = /,;;+«(^Vi - < +1 ) = /"+«(& + i - &)+« 2 (/,;; + i -2/;; 

(20.98) 


Fig. 20.11 (Simulation with 
the leapfrog method) A 
rectangular pulse is 
simulated with the 
two-variable leapfrog 
method. While for a = 1 the 
pulse shape has not changed 
after 1000 steps, for smaller 
values the short wavelength 
components are lost due to 
dispersion 
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To analyze the stability we insert 


fn _ an ikm Ax n _ an ikmAx 

Jm — uc c Um — ce e 

(20.99) 

and obtain the equations 


Gv = v + auil - e~ ikAx ) 

( 20 . 100 ) 

Gu = u + av(e' kAx — 1) + aruilcoskAx — 2) 

( 20 . 101 ) 


which in matrix form read 


(u\ _ ( i + a 2 (2 cos kAx - 2 ) a(e ikAx 
a(l — e~ ikAx ) 1 



( 20 . 102 ) 


The maximum amplification factor G is given by the largest eigenvalue, which is 
one of the roots of 

, t/IcAx\ r. 0 /kAx\ 

(1 — 4a 2 sin" I ^ j — er)(l — a) + 4a 2 sin" ( ^ ) = 0 

(1 - cr + a 2 A 2 )(l -cr)-cr 2 A 2 = 0 (20.103) 


cr = 1 - 2ct 2 sin 2 ± J ~ 2 a 2 sin 2 - 1. (20.104) 

The eigenvalues coincide with those of the two-step method (20.58). 


20.7.2 Lax-Wendroff Scheme 


The Lax-Wendroff scheme can be derived from the Taylor series expansion 


df{t,x) 1 ~,d 2 f{t,x) 

f (t + At, x) = f it, x) H- ——At + -At — 7^2 -f 

dg(t,x) c 2 At 2 d 2 f (t, x) 

= f(t,x) + cAt + — - + ■ • ■ 


dx 


dt 2 


(20.105) 


g{t + At, x) = git, x) + 


dgit, x) 
dt 


At —At 


• d 2 git,x) 
dt 2 


+ ... 


dfit, x) c 2 At 2 d 2 git, x ) 

= git, x) + cAt + —-^ 7 - 7 —- + ... . 


dx 


dt 2 


(20.106) 
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It uses symmetric differences on regular grids (20.45, 20.46) to obtain the iteration 

(20.107) 


f »+1 rn . r . f 9 n m+ 1 - C -1 2 , f 2/m+l + fm- 1 “ 2 /m 


2 Ax 


2Ax 2 


g^ 1 = g n m + cZ\r /, " + !, , /, ”~ 1 + c 2 At 2 9 ' n+l + 9 ' n ~ l ^ 


/f" +1 \ _/1 

W / \ 


2Ax 


+ \M § D 
§ D l + ^-M 


2 Ax 2 


)(?) 


(20.108) 

(20.109) 


with the tridiagonal matrix 


/0 1 

-1 0 1 


D = 


-1 0 1 
- 10 / 

To analyze the stability we insert 


an AkmAx 


rn _ an lkm Ax n _ «« i 

Jm ~ UC c y m — VC C 

and calculate the eigenvalues (compare with 20.102) of 


e 


+ a 2 (cos kAx — 1) iasin^Zix 
ictsin kAx 1 + a 2 (coskAx 


-») 


which are given by 

(7=1+ a 2 (cos kAx — 1) ± \Ja 2 (cos 2 kAx — 1). 
The root is always imaginary and 


\a\ 2 — 1 + (ct 4 — or)(cos kAx — l) 2 < 1 + 4(o; 4 — a 2 ). 


( 20 . 110 ) 


( 20 . 111 ) 


( 20 . 112 ) 


(20.113) 


For ct < 1 we find |cr| < 1. The method is stable but there is wavelength dependent 
damping (Figs. 20.12 and 20.13). 
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Fig. 20.12 (Stability region 
of the Lax-Wendroff 
method) Instabilities appear 
for |a| > 1. In the opposite 
case short wavelength modes 
are damped 



Fig. 20.13 (Simulation with 
the Lax-Wendroff method) 

A rectangular pulse is 
simulated with the 
two-variable Lax-Wendroff 
method. While for a = 1 the 
pulse shape has not changed 
after 2000 steps, for smaller 
values the short wavelength 
components are lost due to 
dispersion and damping 



20.7.3 Crank-Nicolson Scheme 


This method takes the average of the explicit and implicit Euler methods 

fit + At) = fit) + C - (jjL(t, x) + H (t + At, x)) At (20.114) 

git + At) = git) + ^ (^-it,x) + + At,x)^ At (20.115) 

and uses symmetric differences on the regular grids (20.45, 20.46) to obtain 



(20.116) 
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n n+l — n" 4- — ( f" — f n 4- f' !+1 - f”+ 1 't 
-‘m 1 ^ \ 7 /;; +1 7m—1 ' 7 m -f | J m — 1/ 

which reads in matrix notation 

/ f , 1+l w 1 

Uw Vt^wW vfo A&.+i/‘ 

This equation can be solved formally by collecting terms at time t n +\ 

A?/) 1 )\g n+l ) Vf D 1 )\gn) 

and multiplying with the inverse matrix from left 

^>l = / 1 -j d Y( 1 
Uw v-fo i / vf D 1 


(20.117) 


(20.118) 


(20.119) 


( 20 . 120 ) 


Now, if u is an eigenvector of D with purely imaginary eigenvalue A (Sect. 10.3) 


( 1 fzA / U \ /(I ± f A)u\ a ( u \ 

V?*> 1 J{±uJ V(fA±Duj <4 } V±u/ 


and furthermore 


( 1 ~-* D \( u (1:F ? A)U U( 1 t-A)( u 

V-f D 1 )\±u) V(-f A±1 ) u / 4 A \ ±u J 


( 20 . 121 ) 


( 20 . 122 ) 


But, since the eigenvalue of the inverse matrix is the reciprocal of the eigenvalue, 
the eigenvalues of 


/ i / i fzn 

V-f D 1 ) Vf D l ) 


are given by 


l±fA 

iTfA' 


(20.123) 


(20.124) 


Since A is imaginary, we find |tr| = 1. The Crank-Nicolson method is stable and does 
not show damping like the Lax-Wendroff method. However, there is considerable 
dispersion. Solution of the linear system (20.119) is complicated and can be replaced 
by an iterative predictor-corrector method. Starting from the initial guess 




(20.125) 
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we iterate 

/ (0) f„ +I/2 \ 1 / (0) f„ + l\ l/f B W 1 

V (0) g«+‘/2/ 2 V <0) g»+i ) + 2 Vg„ ) V jl 

(^;:)=(y + (i2> |D )fe) 

«n\/f\ / a ri\ / (0)j 

(0) gl+l 


4 J 

D 1 


( 1 4 

VfZ) 1 


/ ( 1 ) f«+l /2 

\ ( 1 ) g/.+V 2 


w ’)(■ ' 

1 J\sJ \J D J[ ( Sn+ 1 / 

:H(iH (»&)-(*) + C 


;) 


(20.126) 

) V 0 , g» + V 2 j 

(20.127) 



»\ + ( !^\ ( 



\ (2, g«+i/ \s 

«/ If D ) 

\ (1) g/i+y2 / 




(20.128) 


In principle this iteration could be repeated more times, but as Teukolsky showed 
[255], two iterations are optimal for hyperbolic equations like the advection or wave 
equation. The region of stability is reduced (Figs. 20.14 and 20.15) compared to the 
implicit Crank-Nicolson method. The eigenvalues are 

(0) cr = 1 ±ia sin kAx | (0) <r| > 1 (20.129) 

a 2 

' 11 o’ = 1 ± ia sin kAx ——sin 2 kAx | (1) er| > 1 (20.130) 


Fig. 20.14 (Simulation with 
the iterated Crank-Nicolson 
method) A rectangular pulse 
is simulated with the 
two-variable iterated 
Crank-Nicolson method. 
Only this method is stable 
for values a > 1 
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Fig. 20.15 (Simulation of a 
triangular pulse) A triangular 
pulse is simulated with 
different two-variable 
methods (dashed curve 
initial conditions, red 
leapfrog, blue 
Lax-Wendroff, green 
iterated Crank-Nicolson). 
This pulse contains less short 
wavelength components than 
the square pulse and shows 
much less deformation even 
after 5000 steps 



l2 , a 2 2 a 3 sin 3 kAx 

’a = 1-sin - kAx ± i(a sin A: Ax-) 

2 4 

,) ct 4 sin 4 A:Z\x a 6 sin 6 A:Ar 

I c2) cr| 2 = 1-+- < 1 for lal < 2. 

1 4 16 - i i - 


(20.131) 


Problems 

Problem 20.1 Waves on a Damped String 

In this computer experiment we simulate waves on a string with a moving boundary 
with the method from Sect. 20.6. 

• Excite the left boundary with a continuous sine function and try to generate stand¬ 
ing waves 

• Increase the velocity until instabilities appear 

• Compare reflection at open and fixed right boundary 

• Observe the dispersion of pulses with different shape and duration 

• The velocity can be changed by a factor n (refractive index) in the region x > 0. 
Observe reflection at the boundary x = 0 

Problem 20.2 Waves with the Fourier Transform Method 

In this computer experiment we use the method from Sect. 20.3 to simulate waves 
on a string with fixed boundaries. 
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• Different initial excitations of the string can be selected 

• The dispersion can be switched off by using u>k = ck instead of the proper eigen¬ 
values (20.31) 

Problem 20.3 Two Variable Methods 

In this computer experiment we simulate waves with periodic boundary conditions. 
Different initial values (rectangular, triangular or Gaussian pulses of different widths) 
and methods (leapfrog, Lax-Wendroff, iterated Crank-Nicolson) can be compared. 


Chapter 21 

Diffusion 


Diffusion is one of the simplest non-equilibrium processes. It describes the transport 
of heat [256, 257] and the time evolution of differences in substance concentrations 
[258]. In this chapter, the one-dimensional diffusion equation 


d 2 

fit, x) = D ^ / (t, x) + S(t, x) 


d_ 

di 


( 21 . 1 ) 


is semi-discretized with finite differences. The time integration is performed with three 
different Euler methods. The explicit Euler method is conditionally stable only for 
small Courant number a — < l /i, which makes very small time steps necessary. 

The fully implicit method is unconditionally stable but its dispersion deviates largely 
from the exact expression. The Crank-Nicolson method is also unconditionally stable. 
However, it is more accurate and its dispersion relation is closer to the exact one. 
Extension to more than one dimension is easily possible, but the numerical effort 
increases drastically as there is no formulation involving simple tridiagonal matrices 
like in one dimension. The split operator approximation uses the one-dimensional 
method independently for each dimension. It is very efficient with almost no loss in 
accuracy. In a computer experiment the different schemes are compared for diffusion 
in two dimensions. 


21.1 Particle Flux and Concentration Changes 

Let /(x, t) denote the concentration of a particle species and J the corresponding 
flux of particles. Consider a small cube with volume h 2 (Fig. 21.1). The change 
of the number of particles within this volume is given by the integral form of the 
conservation law (12.10) 



( 21 . 2 ) 
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Fig. 21.1 Flux through a 
volume element 


J z (z + f) 



where the source term S(r) accounts for creation or destruction of particles due to 
for instance chemical reactions. In Cartesian coordinates we have 


z’ y’’ z '’ - s ( x '’ y 1z ’’ 

/ ^ •/(■*', y',z + ^)~ J z (x y, Z - 


rx+h/2 ry+h/2 rz+h/2 / n 

/ dx' / dV / dz! 1 f '~' 

Jx-h/2 Jy-h/2 J z-h/2 

rx+h/2 ry+h/2 / ^ 

+ / dx' / /, z + -) - j z ( x ' 

Jx-h/2 Jy-h/2 \ 1 

rx+h/2 rz+h/2 / j / 7 \ 

+ / dx' / dz' d v (x', V + z') - /,(*', y - z') 

Jx-h/2 J z-h/2 \ Z Z / 

rz+h/2 ry+h/2 / u r 

+ dz' dy'(j z (x+-,/,z')-J z (x--,y,z'))= 0. (21.3) 

J z-h/2 Jy-h/2 


z')^ = 


2 4 2 
In the limit of small h this turns into the differential form of the conservation law 


h 3 ( ~ r fix, y, z, t) - S(x, y, z, t) )+h 2 ( h + h^p- + j = 0 (21.4) 


)^+h 2 ^ 


d J x dJ z 

-h h —- + h - 

dx dy dz 


or after division by h 3 

+r.fi r, t ) = —divj(r, t) + S(r, f). (21.5) 

dt 

Within the framework of linear response theory the flux is proportional to the gradient 
of/(Fig. 21.2), 


J = ~D grad /. 


( 21 . 6 ) 
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Fig. 21.2 Diffusion due to a 0 0 
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Together we obtain the diffusion equation 




= div(Dgrad/) + S 
at 



(21.7) 

which in the special case of constant D simplifies to 




df 

i = DAf+s ■ 



(21.8) 


21.2 Diffusion in One Dimension 


We will use the finite differences method which works well if the diffusion constant 
D is constant in time and space. We begin with diffusion in one dimension and use 
regular grids t„ = nAt, x m = mAx, f" = f(t„,x m ) and the discretized second 
derivative 


d 2 f f(x + Ax) + f(x — Ax) — 2 f(x) 7 

d^ 2 = - '- — + 0(Ax 2 ) 


(21.9) 


to obtain the semi-discrete diffusion equation 


D 

f(t,x m ) = (f(t,x m+ 1 ) + f(t,x m - f) - 2 f(t,x m )) + S(t,x m ) (21.10) 

or in matrix notation 

f(f) = -^Mf(t) + S(0 (21.11) 


with the tridiagonal matrix 
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Fig. 21.3 (Boundary conditions for 1-dimensional diffusion) Additional boundary points xo, xm+ l 
are used to realize the boundary conditions, (a) Dirichlet boundary conditions the function val- 

o2 . 

ues at the boundary are given /(t,x 0 ) = f 0 (t) i) = (/(x 2 ) - 2f(xi) + £ 0 (t)) 

or fit, x M +i) = 6r+l(0> -7£if(x M ) = (f(x M -i) ~2f(x M ) + £ M+l (t)), (b) Neumann 
boundary conditions the flux through the boundary is given, hence the derivative ^ at the 
boundary /(f, x 0 ) = fit, x 2 ) + 2^- Ji(t) £zf(x i) = (2f{x 2 ) - 2/(xi) + 2^ /i(t)) or 

f(t,x M + 1) = f(t,x M -i)-^jr JmV) jgzf(x M ) = ( 2 f(x M - l) - 2 f(x M ) ~ JmU)), 

(c) No-flow boundary conditions there is no flux through the boundary, hence the deriva- 
tive % = 0 at the boundary f(t,x 0 ) = f(t,x 2 ) J^r/(xt) = (2/(x 2 ) - 2f(x\)) or 

f{t, x M ) = fit, x«- 2 ) tfiXM ) = i-f ( xm - i ) - 2 fix M )) 


1-2 1 \ 
1 -2 1 


M 




1 -2 1 

1 - 2 / 


( 21 . 12 ) 


Boundary conditions can be taken into account by introducing extra boundary 
points xo, xm+i (Fig. 21.3). 
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21.2.1 Explicit Euler (Forward Time Centered Space) 
Scheme 


A simple Euler step (13.3) makes the approximation 

/r 1 - f!n = Htn, x m )At = D^- 2 (/» +1 + - 2/») + S” At. (21.13) 

For homogeneous boundary conditions / = 0 this becomes in matrix form 


//f +1 \ 

//r\ 


/ S'l At \ 


= A : 

+ 


v/m + i J 

\f n M ) 




with the tridiagonal matrix 


A = 


l'~ 2D & Df, 

1 - 20 ^ 


n At 


where a is the Courant number for diffusion 
At 


a = D 


Ax 2 


The eigenvalues of M are (compare 20.30) 


I-. ( kAx 
A = -4 sin 2 I - 


/ kAx\ 

I -I wi 

V 2 ) 


with kAx — 


2tt 


m + r m + r 

and hence the eigenvalues of A are given by 
t kAx 

1 + a A = 1 — 4a sin - ——. 


The algorithm is stable if 
|1 + a A | < 1 for all A 


M 7T 

M + 1 


(21.14) 


+ aM 

(21.15) 

(21.16) 

(21.17) 

(21.18) 

(21.19) 


which holds if 
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-1 < 1 -4asin 2 — < 1. (21.20) 

The maximum of the sine function is sin( 0( ^ | . ) ~ 1. Hence the right hand inequa¬ 
tion is satisfied and from the left one we have 


-1 < 1 - 4a. 


( 21 . 21 ) 


The algorithm is stable for 
At 1 

a = ^ <2 . (21.22) 

The dispersion relation follows from inserting a plane wave ansatz 

e“At = i _ 4 Q sin“ ■ (21.23) 

For a > [ /4 the right hand side changes sign at 


k c Ax = 2arcsin, 



(21.24) 


The imaginary part of uj has a singularity at k :: and the real part has a finite value of 
7 r for k > k c (Fig. 21.4). Deviations from the exact dispersion 

ut = i k 2 (21.25) 


are large, except for very small k. 


Fig. 21.4 (Dispersion of the 
explicit Euler method) The 
dispersion of the explicit 
method is shown for different 
values of the Courant 
number a and compared to 
the exact dispersion (dashed 
curve). The imaginary part 
of u shows a singularity for 
a > V 4 - Above the 
singularity to is complex 
valued 


a 

3 

E 
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21.2.2 Implicit Euler (Backward Time Centered Space) 
Scheme 

Next we use the backward difference 
c +1 - C = f(t n+ ux m )At 
d 2 f 

= T) — ^ (f»+l , X m ) At -{- S(t n +\ , Xni) At 
to obtain the implicit method 

c +l - a + C-\ - 2/- +i ) = c + s? 1 * 

or in matrix notation 

Af„ +1 — f„ + S„ + i At with A = 1 — aM 
which can be solved formally by 
f«+l = A *f„ + A 'Sn+iZff. 

The eigenvalues of A are 


A (A) = 1 + 4a sin" 


kAx 


> 1 


(21.26) 


(21.27) 


(21.28) 


(21.29) 


(21.30) 


and the eigenvalues of A 


-l 


A(A-‘) = A (A)” 1 = - 


1 +4asin 2 ^' 


The implicit method is unconditionally stable since 
|A(A _1 )| < 1. 

The dispersion relation of the implicit scheme follows from 

e uAt = 1 


1 + 4a sin 2 (^§4) 


(21.31) 


(21.32) 


(21.33) 


There is no singularity and u> is purely imaginary. Still, deviations from the exact 
expression are large (Fig. 21.5). 
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Fig. 21.5 (Dispersion of the 
implicit Euler method) The 
dispersion of the fully 
implicit method is shown for 
two different values of the 
Courant number a and 
compared to the exact 
dispersion (dashed curve ) 



Formally a matrix inversion is necessary. Numerically it is much more efficient 
to solve the tridiagonal system of equations (page 75). 

(1 - aM)f(t n+1 ) = f(t„) + S(t n+ \)At. (21.34) 


21.2.3 Crank-Nicolson Method 


The Crank-Nicolson method [259] which is often used for diffusion problems, com¬ 
bines implicit and explicit methods. It uses the Heun method (Sect. 13.5) for the time 
integration 


ftl + 1 

J m 


r = 

J m 


At 

~2 



(in -(-1 •> Xm) T 


df 

dt 



(21.35) 


2 \dx‘ 


(/«+l i 


d 2 f 

,)+ d^ itn 


Xm) ) Xm) S(t n + 1 , X t 


At 


(21.36) 


D At( C +l + /"_!- 2 /" 
2 y Ax 2 


fn +1 I fii+1 _ 7 f 
Jin ' J m —1 Jr, 

Ax 2 


+ 


fn +1 


) 


+ 


o/i . on + 1 


At. 


(21.37) 


This approximation is second order both in time and space and becomes in matrix 
notation 








21.2 Diffusion in One Dimension 


487 


( 




t) f„+i = + 

which can be solved by 


S/i 4“ &n +1 


At 




-> s„ 


3» + l 


(21.38) 


Ar. (21.39) 


Again it is numerically much more efficient to solve the tridiagonal system of 
equations (21.38) than to calculate the inverse matrix. 

The eigenvalues of this method are 


A = 


1 + fM . 


with n — —4 sin" 


kAx 


1-fM 

Since an < 0 it follows 


, a , a 

i + -n<i--n 


and hence 

A < 1. 

On the other hand we have 
1 > -1 


a a 

i + -n>-i + -n 


A > -1. 


e [-4,0], 


(21.40) 

(21.41) 

(21.42) 

(21.43) 

(21.44) 

(21.45) 


This shows that the Crank-Nicolson method is stable [260]. The dispersion follows 
from 


1 -2a sin 2 (^) 
" 1 +2asin 2 (^)' 


For a > '/2 there is a sign change of the right hand side at 


k c Ax — 2arcsin, 



(21.46) 


(21.47) 


The imaginary part of w has a singularity at k, and oj is complex valued for k > k, 
(Fig. 21.6). 
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Fig. 21.6 (Dispersion of the 
Crank-Nicolson method) 
The dispersion of the 
Crank-Nicolson method is 
shown for different values of 
the Courant number a and 
compared to the exact 
dispersion (dashed curve). 
The imaginary part of oj 
shows a singularity for 
a > 1 / 2 . Above the 
singularity lj is complex 
valued. The exact dispersion 
is approached quite closely 
for a ~ 1/2 


B 

< 

3 

B 



21.2.4 Error Order Analysis 

Taylor series gives for the exact solution 

At 2 ■■ At 3 <9 3 

Afexact = At fit, x) + —fit.x) + ' ' ' 

2 6 ot i 

At 2 

= At [Df'it, x) + S{t, a-)] + — [Df'it, x) + Sit, a)] + • • ■ (21.48) 

whereas for the explicit method 
Afexpi = aMf it, x) + Sit, X )At 
At 

= D - - if it, a + Ax) + fit, a — Ax) — 2 fit, a)) + Sit, x)At. 

Ax 2 

At ( , Ax 4 \ 

= Dj^ \Ax 2 fit, a) + —/""a, a) + • • • J + sit, x )At 

DAtAx 2 .... At 2 ■■ 

= Af exac r + - ^2 - f (*’ X ) - *) + ’ ' ’ ( 2L49 ) 

and for the implicit method 

Afimpi = aMf it + At, a) + Sit + At, x)At 
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4“ At, x( At 

At ( 7 Ax 4 .... \ 

/\ / Z\jc^ \ 

+S(t, x)At + D -- ( Ax 1 f”{t, X ) +-*) + •••)+ S(t, x)At 2 

Ax 1 \ 12 / 

At Ax 1 1 

= ^f exact + O X) + -At~f (t,x) + ■ ■ ■ . (21.50) 

The Crank-Nicolson method has higher accuracy in At: 

Afexn, + Af imp i DAtA.X 2 „„ At 3 d 3 f 

AfcN = J p . J p =--- f""(t,x)~ —+ ■•• . (21.51) 

2 12 6 ot j 


21.2.5 Finite Element Discretization 


In one dimension discretization with finite differences is very similar to discretization 
with finite elements, if Galerkin’s method is applied on a regular grid (Chap. 12). The 
only difference is the non-diagonal form of the mass-matrix which has to be applied 
to the time derivative [147]. Implementation of the discretization scheme (12.170) 
is straightforward. The semi-discrete diffusion equation becomes 

<9/1 2 1 \ 

| jJH, x m -i) + -f(t, x m ) + -f(t, x m+ i ) I 

= (fit, x m+ i) + f(t, x m -i) - 2 fit, x m )) + S(t, x m ) (21.52) 

or in matrix form 


('+H f 


D 


m= + 


(21.53) 


This can be combined with the Crank-Nicolson scheme to obtain 
^1 + -M^j (f„ + i — f„) = ( -M 2 f„ + — M 2 f„+i^ H—— (S„ + S„. 


+t) (21.54) 


or 


r, ( x <*\ i 


r, /I a\ 1 

H6-2H 

fn+l = 

[ 1 + ( 6 + 2 j M2 J 


f;i H- — (S» + S„+i) . 


(21.55) 
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21.3 Split-Operator Method for Multidimensions 


The simplest discretization of the Laplace operator in 3 dimensions is given by 

( d 2 d 2 d 2 \ 

Af = {o^ + 3f + a?) f(, ' :, ' y ' z) 

= + M y + f (t ’ x ’ y ’ z ) (21.56) 


where 

1 f(t,x + Ax,y,z) + f(t,x-Ax,y,z)-2f(t,x,y,z) 

"-A?- 

(21.57) 

etc. denote the discretized second derivatives. Generalization of the Crank-Nicolson 
method for the 3-dimensional problem gives 


/(*»+!) = (l - 


a a a 
-Mr - M v - M- 


H 


a a a 

1 + — M V + ~My + ~ M; 


) /( 0 . 

(21.58) 


But now the matrices representing the operators M x , M y , M z are not tridiagonal. To 
keep the advantages of tridiagonal matrices we use the approximations 


t 1 + 2 Mx + 2 My + 2 Mz 


(a a a 

1- M x - M y -M- 

l 2 2 y 2 


and rearrange the factors to obtain 


) “ (‘ + f(‘ + f "■) (‘ + f "■) < 2L59 > 

) ~ (l - \M,) (l - t M .v) (l - f ■ Mz) ( 21 - 60 ) 


f(tn+ 1) = (l - f M,) 1 (l + |m,) (l - |m v ) 1 (l + 2 m v ) (l - |m z ) 1 (l + |m z ) /(»„) 


(21.61) 


which represents successive application of the 1-dimensional scheme for the three 
directions separately. The last step was possible since the operators M, and Mj for 
different directions i ^ j commute. For instance 

M x M y f = M x (f(x, y + Ax) + f(x, y - Ax) - 2 f(x, y)) 

= (/(x + Ax, y + Ay) + f(x — Ax, y + Ax) 

— 2 f(x, y + Ax) + f(x + Ax, y — Ax) 
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+ fix — Ax, y — Ax) — 2f(x,y — Ax) 

-2 f(x + zix, y) - 2 fix - Ax, >') + 4 fix, y)) 
= M y M x f. 


(21.62) 


The Taylor series of (21.58) and (21.61) coincide up to second order with respect to 

aM,\ 

I a a a \ — 1 / a a cr \ 

(. - -M X - -M y - -M Z ) (l + —M x + -My + -M Z ) 

O/^ o t} 

= 1 + a(M x + My + M z ) H—— {M x -t- My + M z ) + Oicx. ) (21.63) 


o ■- o+»(' - (>+<> - >r o - >-) 


a . \ —1 / a , 


a , \ —1 / a 


( q 2 M 2 \( q2 M y\/ a 2 M}\ 

= ( 1 + °M X -\ --— 1 I 1 + aM y H- yy- I 1 1 + aM z -\ --— \ 


+ 0(cC) 


— 1 -t- a(M x + My + M z ) 4- -y- ( M x + My + M z ) 2 4- Oiof). (21.64) 


Hence we have 


= (l + DAt (A+ ^-A 2 + ---\ + ^-iA 2 + ■■•)) 
+ ( 1 + ^ + ...) s -+-. + ^, 


2 

Zir 2 


= fn + AtiDAfn + s„) + —(D-A- + DAS,, + S„) + OiAtAx 1 , AD). 


2 a D) 

(21.65) 


and the error order is conserved by the split operator method. 


Problems 

Problem 21.1 Diffusion in 2 Dimensions 

In this computer experiment we solve the diffusion equation on a two dimensional 
grid for 

• an initial distribution fit = 0, x, y) = 6 x ,oS yt o 

• a constant source fit = 0) = 0, Sf, x, y) = S Xt o(>y,o 

Compare implicit, explicit and Crank-Nicolson method. 








Chapter 22 

Nonlinear Systems 


Nonlinear problems [261, 262] are of interest to physicists, mathematicians and 
also engineers. Nonlinear equations are difficult to solve and give rise to interesting 
phenomena like indeterministic behavior, multistability or formation of patterns in 
time and space. In the following we discuss recurrence relations like an iterated 
function [263] 


x n+1 = f(x n ) (22.1) 

systems of ordinary differential equations like population dynamics models 
[264-266] 

x(t) = f{x, y) 

y(t ) = g(x, y) (22.2) 

or partial differential equations like the reaction diffusion equation [265, 267, 268] 

d ^2 

—c(x, t) = D—c(.x, t ) + /(c) (22.3) 

where f and g are nonlinear in the mathematical sense. 1 We discuss fixed points of 
the logistic mapping and analyze their stability. A bifurcation diagram visualizes 
the appearance of period doubling and chaotic behavior as a function of a control 
parameter. The Ljapunov exponent helps to distinguish stable fixed points and periods 
from chaotic regions. For continuous-time models, the iterated function is replaced by 
a system of differential equations. For stable equilibria all eigenvalues of the Jacobian 
matrix must have a negative real part. We discuss the Lotka-Volterra model, which 
is the simplest model of predator-prey interactions and the Holling-Tanner model, 
which incorporates functional response. Finally we allow for spatial inhomogeneity 
and include diffusive terms to obtain reaction-diffusion systems, which show the 


1 Linear functions are additive f(x + y) = f{x) + f{y) and homogeneous /(ax) = af(x). 
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phenomena of traveling waves and pattern formation. Computer experiments study 
orbits and bifurcation diagram of the logistic map, periodic oscillations of the Lotka- 
Volterra model, oscillations and limit cycles of the Holling-Tanner model and finally 
pattern formation in the diffusive Lotka-Volterra model. 


22.1 Iterated Functions 

Starting from an initial value xq a function / is iterated repeatedly 

xi = fix o) 

X2 = fix l) 

x i+ i = fixi). (22.4) 

The sequence of function values xq,x\ - ■ ■ is called the orbit of xq. It can be visualized 
in a 2-dimensional plot by connecting the points 

ix o,Xi) ->• -»• ixi,x 2 ) -> ix 2 ,x 2 ) -» (Xi,X i+ 1) ix i+ i,x i+ i) 

by straight lines (Fig. 22.1). 


22.1.1 Fixed Points and Stability 

If the equation 

x* = fix*) 

Fig. 22.1 (Orbit of an y 

iterated function) The ‘ 

sequence of points 

ixi,x i+ 1 ), (*/+t, Xi+i) is 

plotted together with the 

curves y = f(x) (dashed) 

and y = x ( dotted ) 


(22.5) 



x 
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Fig. 22.2 (Attractive fixed 
point) The orbit of an 
attractive fixed point 
converges to the intersection 
of the curves y = x and 

y = /(*) 



has solutions x*, then these are called fixed points. Consider a point in the vicinity 
of a fixed point 


X = X* + £q 


( 22 . 6 ) 


and make a Taylor series expansion 


f{x) = f(x* + e 0 ) = fix*) + e 0 fix*) + ■■■=** + £! + ■■■ (22.7) 


with the notation 


£i = £o fix*). 


( 22 . 8 ) 


Repeated iteration gives 2 


f (2 \x) = fifix)) = fix* + £i) H-= x* + £l/'(**) =X* + £2 


fW (x *) = x* + £ n (22.9) 

with the sequence of deviations 

£« = f'ix *)£ n -1 = • • • = (fix*))" e 0 . 

The orbit moves away from the fixed point for arbitrarily small £o if \ f'(x*)\ > 1 
whereas the fixed point is attractive for |/'(.x;*)| < 1 (Fig. 22.2). 

Higher order fixed points are defined by iterating fix) several times. A fixed point 
of order n simultaneously solves 


2 FIere and in the following denotes an iterated function, not a derivative. 
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Fig. 22.3 (Periodic orbit) 
The orbit of an attractive 
fourth order fixed point 
cycles through the values 
*l = / (m),x 2 = fix i), 

X3 = f(X 2 ), X4 = fix 3) 


fix*) / X* 
f (2 \x*) /x* 
f (n ~ l \x*) / X* 

f M (x*) = x*. (22.10) 

The iterated function values cycle periodically (Fig. 22.3) through 

x* f(x*) -* / (2) (x*) • ■ ■ / (,!_1) (x*). 

This period is attractive if 

\fix*) f'ifix*)) fif (2 \x*)) ■ ■ ■ fif n ~ l) ix*))\ < 1. 



22.1.2 The Ljapunov-Exponent 

Consider two neighboring orbits with initial values xo and xo + £o ■ After n iterations 
the distance is 

!/(/(• • • fix o))) - /(/(• • • fix o + £o))) I = \e 0 \e Xn (22.11) 

with the so called Ljapunov-exponent [269] A which is useful to characterize the 
orbit. The Ljapunov-exponent can be determined from 

A = lim - In / l/ W ^o + £ o)-/ ( ” ) ^o)l \ 

"->■ °° n V l £ ol / 


( 22 . 12 ) 
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or numerically easier with the approximation 


I f(xo + So) - /Oo)l = |£oll/ , Oo)| 


l/(/Oo + So)) - f(f(xo))\ = | (f(x 0 + £ 0 ) - f(x 0 ))\\f(x 0 + £ 0 )l 

= \so\\f'(xo)\\f'(x 0 + e 0 )\\ 


(22.13) 


\f (n \x 0 +e 0 )- f in Hx 0 )\ = \so\\f(x 0 )\\f(x l )\---\f(x n . l )\ (22.14) 


from 



(22.15) 


For a stable fixed point 


A—> In |/'(**) I <0 


(22.16) 


and for an attractive period 


A In | f(x*) /'(/(**) • • • /'(/ ( " _1) (**))! < 0. 




(22.17) 


Orbits with A < 0 are attractive fixed points or periods. If, on the other hand, A > 0, 
the orbit is irregular and very sensitive to the initial conditions, hence is chaotic. 


22.1.3 The Logistic Map 

A population of animals is observed yearly. The evolution of the population density 
N is described in terms of the reproduction rate r by the recurrence relation 


(22.18) 


Nn+l — r Nn 


where N„ is the population density in year number n. If r is constant, an exponential 
increase or decrease of N results. 

The simplest model for the growth of a population which takes into account that 
the resources are limited is the logistic model by Verhulst [270]. He assumed that the 
reproduction rate r depends on the population density A in a simple way (Fig. 22.4) 



(22.19) 
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Fig. 22.4 (Reproduction 
rate of the logistic model) At 
low densities the growth rate 
has its maximum value ro . At 
larger densities the growth 
rate declines and reaches 
r = 0 for N = K. The 
parameter K is called 
carrying capacity 


r (N) 



The Verhulst model (22.19) leads to the iterated nonlinear function 

N n +i = r 0 N n - (22.20) 

K 

with ro > 0, K >0. We denote the quotient of population density and carrying 
capacity by the new variable 

Xn = 4 N„ (22.21) 

K 

and obtain an equation with only one parameter, the so called logistic mapping 
1 1 ( N n \ 

x n + 1 = — N n+ \ = — r 0 N„ ( 1 - — j = r 0 ■ x n ■ (1 - x n ). (22.22) 


22.1.4 Fixed Points of the Logistic Map 


Consider an initial point in the interval 

0 < xq < 1 . 


(22.23) 


We want to find conditions on r to keep the orbit in this interval. The maximum value 
of x n+ i is found from 


d-T; ; ] 
dx„ 


= r( 1 — 2x„) = 0 


(22.24) 


which gives x„ = */2 and max(x /!+ i) = r/4. If r > 4 then negative x n appear after 
some iterations and the orbit is not bound by a finite interval since 
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l*n+ll 

\x n \ 


= kld + l^l) > 1. 


The fixed point equation 


*2 


(22.25) 


(22.26) 


always has the trivial solution 

x* = 0 (22.27) 

and a further solution 

1 

x* = 1 - - (22.28) 

r 

which is only physically reasonable for r > 1, since x should be a positive quantity. 
For the logistic mapping the derivative is 

fix) = r-2rx (22.29) 

which for the first fixed point x* = 0 gives |/'(0)| = r. This fixed point is attractive 
forO < r < 1 and becomes unstable for r > 1. For the second fixed point we have 
|/'(1 — i)| = 12 — r|, which is smaller than one in the interval 1 < r < 3. For r < 1 
no such fixed point exists. Finally, for r\ = 3 the first bifurcation appears and higher 
order fixed points become stable. 

Consider the fixed point of the double iteration 

x* = r(r(x* - x* 1 ) - r 2 (x* - x* 2 ) 2 ). (22.30) 


All roots of this fourth order equation can be found since we already know two of 
them. The remaining roots are 




2 r 


M.2 


(22.31) 


They are real valued if 

(r — l) 2 — 4 > 0 —» r > 3 (or r < — 1). 


(22.32) 


For r > 3 the orbit oscillates between x* and Xj until the next period doubling 
appears for ri = 1 + V6. With increasing r more and more bifurcations appear and 
finally the orbits become chaotic (Fig. 22.5). 
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0.4 

Fig. 22.5 (Orbits of the logistic map) Left For 0 < r < 1 the logistic map has the attractive fixed 
point x* = 0. Middle In the region 1 < r < 3 this fixed point becomes unstable and another stable 
fixed point is at x* = 1 — 1 / r. Right For 3 < r < 1 + \/6 the second order fixed point (22.31) is 
stable. For larger values of r more and more bifurcations appear 






Fig. 22.6 (Bifurcation diagram of the logistic map) For different values of r the function is iterated 
1100 times. The first 1000 iterations are dropped to allow the trajectory to approach stable fixed 
points or periods. The iterated function values xiooo • • • xnoo 316 plotted in a r-x diagram together 
with the estimate (22.15) of the Ljapunov exponent. The first period doublings appear at r = 3 and 
r = 1 + \/6. For larger values chaotic behavior is observed and the estimated Ljapunov exponent 
becomes positive. In some regions motion is regular again with negative Ljapunov exponent 


22.1.5 Bifurcation Diagram 

The bifurcation diagram visualizes the appearance of period doubling and chaotic 
behavior as a function of the control parameter r (Fig. 22.6). 
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22.2 Population Dynamics 


If time is treated as a continuous variable, the iterated function has to be replaced by 
a differential equation 


d N 

d7 = 


or, more generally by a system of equations 



( Nx \ 


(MNi- 

■N n )\ 

d 

dr 

Nn 

= 

fi(Ni ■ 

■N n ) 


\ ) 


yMNi- 

■N n )j 


22.2.1 Equilibria and Stability 


(22.33) 


(22.34) 


The role of the fixed points is now taken over by equilibria, which are solutions of 


0 = — = f{N eq ) (22.35) 

at 

which means roots of /(IV)- Let us investigate small deviations from equilibrium 
with the help of a Taylor series expansion. Inserting 

N = N eq +£ (22.36) 

we obtain 

^ = f(N eq ) + f'(N eq )^+--- (22.37) 

but since f(N eq ) — 0, we have approximately 

= f'WegK (22.38) 

dr 

with the solution 

£(f) = £ 0 exp{/'(lV e? )r}. (22.39) 


The equilibrium is only stable if Tie f'(N eq ) < 0, since then small deviations 
disappear exponentially. For (He f'{N eq ) > 0 deviations will increase, but the 
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exponential behavior holds only for not too large deviations and saturation may 
appear. If the derivative f'(N eq ) has a nonzero imaginary part then oscillations will 
be superimposed. For a system of equations the equilibrium is defined by 


//iovr- 

■Nf)\ 


/°\ 

h(NT ■ 

• N e n q ) 

— 

0 

kMN?- 

■ K q )) 


U/ 


and if such an equilibrium exists, linearization gives 


l N A 


/N?\ 


/6\ 

n 2 

= 

N eq 

+ 

6 

UJ 


\Nl q , 


UJ 


M. 

9N„ 

dfc 

dN„ 


M. 

dN„ 



/M 


d 

6 


dt 




dfi dfi 
ON i dN 2 
dh_ dh 
dNi dN2 


dfn dfn 

\ dNi dN2 


\ 

/ 


/ 6 \ 

6 

W 


(22.40) 


(22.41) 


(22.42) 


The equilibrium is stable if all eigenvalues A,- of the derivative matrix have a negative 
real part. 


22.2.2 The Continuous Logistic Model 

The continuous logistic model describes the evolution by the differential equation 

—- = ro.x(l — x). (22.43) 

dt 

To find possible equilibria we have to solve 

x eq ( l-x eq ) = 0 (22.44) 


which has the two roots x eq = 0 and x eq = 1 (Fig. 22.7). 

The derivative f' is 

f(x) = A (r 0 x( 1 - x)) = r 0 (l - 2x). (22.45) 

dx 

Since /'(0) = ro > 0 and /'(1) = —ro < 0 only the second equilibrium is stable. 
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Fig. 22.7 (Equilibria of the 
logistic model) The 
equilibrium x eq = 0 is 
unstable since an 
infinitesimal deviation grows 
exponentially in time. The 
equilibrium x eq = 1 is stable 
since initial deviations 
disappear exponentially 



22.3 Lotka-Volterra Model 

The model by Lotka [271] and Volterra [272] is the simplest model of predator-prey 
interactions. It has two variables, the density of prey (H) and the density of predators 
(P). The overall reproduction rate of each species is given by the difference of the 
birth rate r and the mortality rate m 

dN 

- = (r — m)N 

d t 

which both may depend on the population densities. The Lotka-Volterra model 
assumes that the prey mortality depends linearly on the predator density and the 
predator birth rate is proportional to the prey density 


m H = aP rp = bH 


(22.46) 


where a is the predation rate coefficient and b is the reproduction rate of predators per 
1 prey eaten. Together we end up with a system of two coupled nonlinear differential 
equations 

d H 

— = f(H , P) = r H H — aHP 
d t 


d P 

~d7 


= g(H, P ) = bHP - m P P 


(22.47) 


where r H is the intrinsic rate of prey population increase and m p the predator mor¬ 
tality rate. 
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22.3.1 Stability Analysis 

To find equilibria we have to solve the system of equations 


/(//, P) = r H H — aHP = 0 

g(H,P) = bHP -mpP = 0. (22.48) 

The first equation is solved by H eq — 0 or by P eq = >'h /«■ The second equation is 
solved by P eq — 0 or by H eq — mp /b. Hence there are two equilibria, the trivial one 


P eq — Pt eq = 0 


(22.49) 


and a nontrivial one 


rn m p 

P = — H = —— 

1 eq — ±± eq — , 

a b 


Linearization around the zero equilibrium gives 


d H d P 

— = r H H H- — = -nipP 

At At 


(22.50) 


(22.51) 


This equilibrium is unstable since a small prey population will increase exponentially. 
Now expand around the nontrivial equilibrium: 


P = P eq + H= H a 


V 


dV df df am P 

— = 0-h ~ aP eq )v - aH eq i = - 


dg dg 


b 
br H 

mpK = - 1) 


or in matrix notation 


d 

dr 




The eigenvalues are purely imaginary 
A = Azi^/mnrp = ±itu 


(22.52) 

(22.53) 

(22.54) 

(22.55) 

(22.56) 


and the corresponding eigenvectors are 
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Fig. 22.8 (Lotka Volterra model) The predator and prey population densities show periodic oscil¬ 
lations (Right). In the H-P plane the system moves on a closed curve, which becomes an ellipse for 
small deviations from equilibrium (Left) 


( i ( am P /b\ 

\ br H /a ) ’ \iJm H r P ) ' 


(22.57) 


The solution of the linearized equations is then given by 
b I rp 

4 (f) = 4 o cos tot H— -? 7 o sin u>t 

a V niH 

a lm h 

r](t) — rj 0 cost ot -- 4 0 sin cut 

b V rp 

which describes an ellipse in the £ — rj plane (Fig. 22.8). The nonlinear equations 
(22.48) have a first integral 


(22.58) 


rn In P(t) — a P(t) — b Hit) + m P In H(t) = C 


(22.59) 


and therefore the motion in the H — P plane is on a closed curve around the equi¬ 
librium which approaches an ellipse for small amplitudes £, 77 . 


22.4 Functional Response 

Holling [273, 274] studied predation of small mammals on pine sawflies. He sug¬ 
gested a very popular model of functional response. Holling assumed that the preda¬ 
tor spends its time on two kinds of activities, searching for prey and prey handling 
(chasing, killing, eating, digesting). The total time equals the sum of time spent on 
searching and time spent on handling 

7^ — Psearch T 7 handling- 


(22.60) 
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Fig. 22.9 Functional 
response of Fiolling’s model 



Capturing prey is assumed to be a random process. A predator examines an area 
a per time and captures all prey found there. After spending the time T s ,, arc j, the 
predator examined an area of aT searc h and captured Hj — HaT searc h prey. Hence 
the predation rate is 


H r 

~HT 


a- 


* search 


1 


a- 


1 "F ^handling/ T sea rch 


(22.61) 


The handling time is assumed to be proportional to the number of prey captured 


^handling — TgH C%T searc h 


(22.62) 


where 7), is the handling time spent per one prey. The predation rate then is given by 


a 

1 T ocHTg 


(22.63) 


At small densities handling time is unimportant and the predation rate is ao = a 
whereas at high prey density handling limits the number of prey captured and the 
predation rate approaches aoo — -jjj; (Fig. 22.9). 


22.4.1 Holling-Tanner Model 

We combine the logistic model with Holling’s model for the predation rate [273-275] 
dH ( H \ ( H \ a 

~dt = ' hH v - aHP = ' hH Yh) ~ T^aHT), HP = f (H ’ P) 

(22.64) 
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and assume that the carrying capacity of the predator is proportional to the density 
of prey 


dP 

dt 




— )=g(H, P). 


(22.65) 


Obviously there is a trivial equilibrium with P eq = H eq — 0. Linearization gives 


d H d P 

-=r H H-\ - —— —r P P 

dt dt 


( 22 . 66 ) 


which shows that this equilibrium is unstable. There is another trivial equilibrium 
with P eq = 0, H eq = Kh- After linearization 


P = £+... H = K H + ri+... 
we find 

drt Kh + 77 a 

— = r H (K H + 77)0 ----)- 

dr K h 1 + a(K H + r,)T h 


(22.67) 


(K h + ti)Z+... 


a 


~r H r\ - 


1 + aK H Th 


K h £ 


( 22 . 68 ) 


~7~ = r PC 

dt 

The eigenvalues of the linearized equations 


OH 


>'H - 


l+aK H T,, 


K, 


rp 


')(?) 


(22.69) 


(22.70) 


A = YH y TP ± \s/(r H - r P ) 2 — r H , 


rp- 


(22.71) 


Let us now look for nontrivial equilibria. The nullclines (Fig. 22.10) are the curves 
defined by = 0 and = 0 , hence by 


P= — (\-^-]{\ + aHT h ) 
a V k h) 


(22.72) 


P = kH. 


(22.73) 
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Fig. 22.10 Nullclines of the 
predator prey model 



The H-nullcline is a parabola at 


H,„ 


°*Th K h 
2aT h K H l m 


0 uT h + K h 1 ) 2 
4 aT h K~ l 


(22.74) 


It intersects the H-axis at H = K H and H = — 1 /a.T h and the P-axis at P = r H /a. 
There is one intersection of the two nullclines at positive values of H and P which 
corresponds to a nontrivial equilibrium. The equilibrium density H eq is the positive 
root of 


r H OiT h H~ q + (r H + ak:K H — r H K H aT h ) H eq — r H K H = 0. 

It is explicitly given by 

r H + akK H — r H K H aTh 


H eq = 


2r H oiT h 

J ( r H + akK H — r H K H aTi,) 2 + Ar H aThr H K H 


2r H aT h 

The prey density then follows from 


(22.75) 


(22.76) 


P — H k 
1 eq — 


(22.77) 


The matrix of derivatives has the elements 


m H p = 


m H H = 


m PP = 


df_ 

dP 

df_ 

dH 

dg 

dP 


aH e , 


1 + aTi,H eq 


akH e , 


+ 


u z H e z q kT h 


1 + aT h H eq (1 + aT/,H eq ) 2 


= = ~r P 
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9g , 

m PH = =rpk 
oH 


from which the eigenvalues are calculated as 


(22.78) 


A = 


m hh + m PP 
2 


± 


I (m HH + mpp ) 2 ~ ~ 

Y-^-( m HH m PP - m HP m PH ). 


(22.79) 


Oscillations appear, if the squareroot is imaginary (Fig. 22.1 1). 


22.5 Reaction-Diffusion Systems 

So far we considered spatially homogeneous systems where the density of a popula¬ 
tion, or the concentration of a chemical agent, depend only on time. If we add spatial 
inhomogeneity and diffusive motion, new and interesting phenomena like pattern 
formation or traveling excitations can be observed. 


22.5.1 General Properties of Reaction-Diffusion Systems 


Reaction-diffusion systems are described by a diffusion equation 3 where the source 
term depends non-linearly on the concentrations 


d_ 

dt 


( Cl \ 


( D > 

\ 


! ci \ 


= 



A 




\ 

Dn ) 


\ c */ 


*i«c})\ 


(22.80) 


22.5.2 Chemical Reactions 

Consider a number of chemical reactions which are described by stoichiometric 
equations 

= 0. (22.81) 
i 


3 We consider only the case, that different species diffuse independently and that the diffusion 
constants do not depend on direction. 













population density population density population density 
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Fig. 22.11 (Holling-Tanner model) Top evolution from an unstable equilibrium to a limit cycle. 
Middle a stable equilibrium is approached with oscillations. Bottom stable equilibrium without 
oscillations 
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The concentration of agent A, is 

c, = Cj, 0 + q* 

with the reaction variable 

t' q 0 
x = - 


(22.82) 


(22.83) 


and the reaction rate 

dx 1 dc,- 
dr v; dr 




(22.84) 


which, in general is a nonlinear function of all concentrations. The total concentration 
change due to diffusion and reactions is given by 


d 


dt 


Ck 


D k A c k + Vkjrj = D k A c k + F k ({c,}). 
i 


(22.85) 


22.5.3 Diffusive Population Dynamics 

Combination of population dynamics (22.2) and diffusive motion gives a similar set 
of coupled equations for the population densities 

%-N k = D k A N k + f k (N i, N 2 , ■ ■ ■ N„). (22.86) 


22.5.4 Stability Analysis 


Since a solution of the nonlinear equations is not generally possible we discuss small 
deviations from an equilibrium solution /V[' 74 with 

i-N k = AN k = 0. (22.87) 

at 

Obviously the equilibrium obeys 


fANi ■ ■ ■ N n ) = 0 k = 1,2 • • • n. 


( 22 . 88 ) 


4 We assume tacitly that such a solution exists. 
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We linearize the equations by setting 

N k = N e k q +£ k 

and expand around the equilibrium 


(22.89) 


d 

/6\ 

6 


( Dl 

\ 

Ft 






UJ 


\ 

Dn / 


/A6\ 

A £ 2 


dh 

8/i 

dN\ 

dNn 

dfi 

dfi 

dNi 

dNi 

Of, 

dh_ 

dNi 

dN 2 


M \ 

dN, 

dh 

ON,, 



/ 6 \ 

6 

UJ 


(22.90) 


Plane waves are solutions of the linearized problem.'’ Using the ansatz 


= 0 , « e 


kx) 


we obtain 



/fc\ 

6 


/6\ 

6 


/6\ 

6 

iuj 

UJ 

= -k 2 D 

UJ 

+ Mq 

UJ 


(22.91) 


(22.92) 


where Mq denotes the matrix of derivatives and D the matrix of diffusion constants. 
For a stable plane wave solution A = iu> is an eigenvalue of 

M k = M 0 -k 2 D (22.93) 


with 

91(A) < 0. (22.94) 

If there are purely imaginary eigenvalues for some k they correspond to stable solu¬ 
tions which are spatially inhomogeneous and lead to formation of certain patterns. 
Interestingly, diffusion can lead to instabilities even for a system which is stable in 
the absence of diffusion [276]. 


5 Strictly this is true only for an infinite or periodic system. 
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22.5.5 Lotka Volterra Model with Diffusion 


As a simple example we consider again the Lotka Volterra model. Adding diffusive 
terms we obtain the equations 


d_ ( H' 
dt 


r H H — aHP\ , ( D H 


pW'.- . 

\P J \bHP -m P P ) 
There are two equilibria 




H eq = P eq = 0 


and 


yh mp 

p — JL h — __ 

a b 


The Jacobian matrix is 


-aH, 


Mo dc F (C 0 ) ( H bPeci eq bHeq _ mp 


eq 


which gives for the trivial equilibrium 
'r H -D H k 2 0 


M k = 


0 


-m p — Dpk~ 


) 


(22.95) 


(22.96) 


(22.97) 


(22.98) 


(22.99) 


One eigenvalue Ai = —m p — Dpk 1 is negative whereas the second \ 2 = rn — Dnk 2 
is positive for k 2 < /'/-/ / F // • Hence this equilibrium is unstable against fluctuations 
with long wavelengths. For the second equilibrium we find: 


M k = 


(~D H k 2 - \ 

V b -f -Dpk 2 ) 


( 22 . 100 ) 


tr (. M k ) = -{D h + D P )k 2 


det (Mk) = m P r H + D H D P k A 

A =-— -k 2 ± -f(D H - D P ) 2 k 4 - 4m P r H . (22.101) 

For small k with k 2 < 2^Jmpr H /\D H — Dp damped oscillations are expected 
whereas the system is stable against fluctuations with larger k (Figs. 22.12, 22.13 
and 22.14). 
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t=1.3 t=6.4 t=30 t=814 



x 


Fig. 22.12 (Lotka-Volterra model with diffusion) The time evolution is calculated for initial random 
fluctuations. Colors indicate the deviation of the predator concentration P(x , y, t ) from its average 
value (blue'. A P < —0.1, green : —0.1 < A P < —0.01, black: —0.01 < A P < 0.01, yellow: 
0.01 < A P < 0.1, red : A P > 0.1). Parameters as in Fig. 22.13 


Fig. 22.13 (Dispersion of 
the diffusive Lotka-Volterra 
model) Real (full curx’e) and 
imaginary part (broken line) 
of the eigenvalue A (22.101) 
are shown as a function of k. 
Parameters are 
Dh = D P = 1, 
mp=t'H=a = b = 0.5 



Problems 

Problem 22.1: Orbits of the Iterated Logistic Map 

This computer example draws orbits (Fig. 22.5) of the logistic map 

x„+i = r 0 ■ x„ ■ (1 - x n ). (22.102) 

You can select the initial value xo and the variable r. 

Problem 22.2: Bifurcation Diagram of the Logistic Map 

This computer example generates a bifurcation diagram of the logistic map (Fig. 22.6). 
You can select the range of r. 

Problem 22.3: Lotka-Volterra Model 

Equation (22.47) are solved with the improved Euler method (Fig. 22.8). The pre¬ 
dictor step uses an explicit Euler step to calculate the values at t + At/2 
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Fig. 22.14 (Traveling waves in the diffusive Lotka-Volterra model) Initially P(x, y) = P elj and 
H(x,y) is peaked in the center. This leads to oscillations and a sharp wavefront moving away from 
the excitation. Color code and parameters as in Fig. 22.12 


At 


At 


H pr (t +—) = H(t) + (r H H(t) - aH(t)P{t )) — 


2 

At 


Pprit + —) = P{t) + (bH(t)P(t) - m p P(t)) 
and the corrector step advances time by At 


2 

At 

2 


(22.103) 

(22.104) 


H(t + At) — H(t) + (rnHprit H——) — aH pr {t H— —)P pr (t H —— m At 


(22.105) 
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P(t + A t) — P(t) + (bH pr (t + —)P pr (t + —) — m„P 


P 1 pr 


At ' 
(t+—) 


f At. 

(22.106) 


Problem 22.4: Holling-Tanner Model 


The equations of the Holling-Tanner model (22.64), (22.65) are solved with the 
improved Euler method (see Fig. 22.11). The predictor step uses an explicit Euler 
step to calculate the values at t + At/2: 


H pr (t + —) = H(t) + — 

Ppr (t +Y )= P(t) + 9(H(t) ’ P(t)) Y 


(22.107) 

(22.108) 


and the corrector step advances time by At: 

Hit + At) = Hit) + fiH pr it + p pr it + y» Ai (22.109) 

Pit + At) = Pit) + giH pr it + y)’ Pprit + Y ))At - (22.110) 

Problem 22.5: Diffusive Lotka-Volterra Model 


The Lotka-Volterra model with diffusion (22.95) is solved in 2 dimensions with 
an implicit method (21.2.2) for the diffusive motion (Figs. 22.12 and 22.14). The 
split operator approximation (21.3) is used to treat diffusion in x and y direction 
independently. The equations 


(Hit + At)\ (A~ l H{t)\ ( A- 1 f (Hit), Pit)) At \ 

V P(t + A n) V A ~'P(t) ) \ A- l g(H(t), Pit))At ) 

„ (A- 1 a; 1 [Hit) + f(H(t), P(f))Af]\ 

~ V A r lA 7‘ [Pit) + g(H(t), P(t))At) ) 1 j 

are equivalent to the following systems of linear equations with tridiagonal 
matrix (5.3): 


A y U = H(t) + f(H(t), Pit))At (22.112) 

U = A x H(t + At) (22.113) 


A y V = Pit) + g(H(t), P(t))At (22.114) 

V = A x P(t + At). (22.115) 


Periodic boundary conditions are implemented with the method described in Sect. 5.4. 


Chapter 23 

Simple Quantum Systems 


In this chapter we study simple quantum systems. A particle in a one-dimensional 
potential V (x) is described by a wave packet which is a solution of the partial 
differential equation [277] 



(23.1) 


We discuss two approaches to discretize the second derivative. Finite differences 
are simple to use but their dispersion deviates largely from the exact relation, except 
high order differences are used. Pseudo-spectral methods evaluate the kinetic energy 
part in Fourier space and are much more accurate. The time evolution operator can 
be approximated by rational expressions like Cauchy’s form which corresponds to 
the Crank-Nicholson method. These schemes are unitary but involve time consuming 
matrix inversions. Multistep differencing schemes have comparable accuracy but are 
explicit methods. Best known is second order differencing. Split operator methods 
approximate the time evolution operator by a product. In combination with finite 
differences for the kinetic energy this leads to the method of real-space product 
formula which can be applied to wavefunctions with more than one component, for 
instance to study transitions between states. In a computer experiment we simulate 
a one-dimensional wave packet in a potential with one or two minima. 

Few-state systems are described with a small set of basis states. Especially the 
quantum mechanical two-level system is often used as a simple model for the tran¬ 
sition between an initial and a final state due to an external perturbation .' Its wave- 
function has two components 



(23.2) 


which satisfy two coupled ordinary differential equations for the amplitudes C\p, of 
the two states 


l For instance collisions or the electromagnetic radiation field. 
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<(c;M££)(£) 

In several computer experiments we study a two-state system in an oscillating 
field, a three-state system as a model for superexchange, the semiclassical model and 
the Landau—Zener model for curve-crossing and the ladder model for exponential 
decay. The density matrix formalism is used to describe a dissipative two-state system 
in analogy to the Bloch equations for nuclear magnetic resonance. In computer 
experiments we study the resonance line and the effects of saturation and power 
broadening. Finally we simulate the generation of a coherent superposition state or 
a spin flip by applying pulses of suitable duration. This is also discussed in connection 
with the manipulation of a Qubit represented by a single spin. 


23.1 Pure and Mixed Quantum States 

Whereas pure states of a quantum system are described by a wavefunction, mixed 
states are described by a density matrix. Mixed states appear if the exact quantum 
state is unknown, for instance for a statistical ensemble of quantum states, a system 
with uncertain preparation history, or if the system is entangled with another system. 
A mixed state is different from a superposition state. For instance, the superposition 

IV >= ColVo > +Ci\ipi > (23.4) 

of the two states IVo > and |Vi > is a pure state, which can be described by the 
density operator 

IV >< ip\ = |c 0 | 2 lVo >< Vo I + |Ci| 2 iVi >< Vil 
+ CoC*\ipo >< V’iI + C* 0 c l IVh >< Vo I (23.5) 

whereas the density operator 

P = FolVo >< Vol + PilVi >< V’iI (23.6) 

describes the mixed state of a system which is in the pure state | Vo > with probability 
po and in the state | V’l > with probability pi = 1 — po- The expectation value of an 
operator A is in the first case 

< A >=< V’lAIV’ >= |C 0 | 2 < V’olAIV’o > +|Ci | 2 < V’lAIVh > 

+ C 0 C* < V’l I A| Vo > +C* 0 C j < VolAIVi > (23.7) 

and in the second case 


< A >= po < V’oI A | V>o > +P\ < V’l I A | V’l > • 


(23.8) 
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Both can be written in the form 

< A >= tr( pA). (23.9) 

23.1.1 Wavefunctions 

The time evolution of a quantum system is governed by the time dependent 
Schroedinger equation [278] 

>= H\ip > (23.10) 

at 

for the wavefunction if}. The brackets indicate that \i/j > is a vector in an abstract 
Hilbert space [47], Vectors can be added 

\ip >= IVh > +1^2 >= \4>i + V>2 > (23.11) 

and can be multiplied with a complex number 

\ip >= \\ipi >= |A-01 > . (23.12) 

Finally a complex valued scalar product of two vectors is defined 2 

c = < VhlV* > (23.13) 

which has the properties 

< 4>i\ip2 >=< VhlVh >* (23.14) 

< Vh|AV>2 >= A < ipi\ip 2 >=< A*'*/hlV’2 > (23.15) 

< + ip 2 >—< V’lV’t > + < V #2 > (23.16) 

< 4>\+ i>2\ip >=< i’lli’ > + < ipiltp > ■ (23.17) 


defined by < i/’ n \ip n , >= J d 3 r x ■ ■ • d 3 r N ip*(ri ■ ■ ■ r N )^ n ,(n ■ ■ ■ r N ). 
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23.1.2 Density Matrix for an Ensemble of Systems 

Consider a thermal ensemble of systems. Their wave functions are expanded with 


respect to basis functions \ip s > as 

W >=X<m>. (23.18) 

S 

The ensemble average of an operator A is given by 

< A > = < i/iAf > = < X C$i> s AC S 'ip S ' > (23.19) 

S,S f 

= ^CfC^A ss ~ = tr (pA) (23.20) 

s,s' 

with the density matrix 

= (23-21) 

s,s' 

The wave function of an /V-state system is a linear combination 

\i/j >= Ci|t/>i > +C 2 \i>2 > H- C N \tp N > . (23.22) 

The diagonal elements of the density matrix are the occupation probabilities 

Pit = TcTF P22=IGP--- Pnn = JcivP (23.23) 

and the non diagonal elements measure the correlation of two states 3 

Pi2 = P 21 = C*C~u ■■■ ■ (23.24) 


23.1.3 Time Evolution of the Density Matrix 

The expansion coefficients of 

iv>>=X c *i^ > (23 - 25 ) 

s 

can be obtained from the scalar product 

C s =< tp s \il> > . 

3 They are often called the “coherence” of the two states. 


(23.26) 
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Hence we have 

C*c s , = < \4> s >< Vvl>=< VvIV’ >< V’lV’s > (23.27) 

which can be considered to be the s', s matrix element of the projection operator 


IV> >< V’l 

C S *Q = ( (23.28) 

The thermal average of |t/t >< ip\ is the statistical operator 

p = \ip >< ip\ (23.29) 

which is represented by the density matrix with respect to the basis functions | ip s > 

Ps’s = W >< ip\ S ' S = CfC;. (23.30) 

From the Schroedinger equation 

ih\i) >= H\ip > (23.31) 

we find 

-ih < tp\ =< Hip | =< %j)\H (23.32) 

and hence 

ihp = ih(\ip >< ij)\ + | ij) >< = |Hip >< ip\ — |i/; >< Hip |. (23.33) 


Since the Hamiltonian H is identical for all members of the ensemble we end up 
with the Liouville-von Neumann equation 

iHp — Hp — pH = [H, p\. (23.34) 

With respect to a finite basis this becomes explicitly: 

ihpa = Hjjpjj — PijHji = Hijpji — PijHjj (23.35) 

j Hi 

ihpik — 'y HijPjk PijHjk 
j 

= (Hu - H kk )p ik + H ik (p kk - pa) + ^ (HijPjk - PijH jk ). (23.36) 

j&.k 
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23.2 Wave Packet Motion in One Dimension 

A quantum mechanical particle with mass m p in a one-dimensional potential V (x) 
(Fig. 23.1) is described by a complex valued wavefunction ip(x). We assume that the 
wavefunction is negligible outside an interval [a, b]. This is the case for a particle 
bound in a potential well i.e. a deep enough minimum of the potential or for a 
wave-packet with finite width far from the boundaries. Then the calculation can be 
restricted to the finite interval [a, b] by applying the boundary condition 

ip(x) — 0 forx < a or x > b (23.37) 

or, if reflections at the boundary should be suppressed, transparent boundary condi¬ 
tions [279], 

All observables (quantities which can be measured) of the particle are expectation 
values with respect to the wavefunction, for instance its average position is 



■b 


(23.38) 


The probability of finding the particle at the position .vq is given by 


P(x = x 0 ) = mx 0 )\ 2 . 


(23.39) 


For time independent potential V (x) the Schroedinger equation 



(23.40) 


can be formally solved by 


Fig. 23.1 Potential well 



a 


b 
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V>(0 = U(t, t 0 )ip(t 0 ) = exp 


i (t ~ t 0 ) 


H 




(23.41) 


If the potential is time dependent, the more general formal solution is 


'ip(t) = U 0f, t 0 )ip(t 0 ) = T, exp 

00 1 / • \ . 

5 ^) 


H(r)dr 


'to 


V’(fo) 


= 2 . dt ' [ o du ---- dt » Tt {H(h)H(t 2 )... H(t n )} 


(23.42) 


where T, denotes the time ordering operator. The simplest approach for discretization 
is to divide the time interval 0 ... t into a sequence of smaller steps 


U ( t, t 0 ) — U(t, fjv-i)... U fe, h)U (?i, t 0 ) 


(23.43) 


and to neglect the variation of the Hamiltonian during the small interval At — t n+ \ —t n 
[280] 


U ( t n+u t n ) = exp 



(23.44) 


23.2.1 Discretization of the Kinetic Energy 


The kinetic energy 


T^(x) = 


H 2 d 2 
2m p dx 2 


ip(x) 


(23.45) 


is a nonlocal operator in real space. It is most efficiently evaluated in Fourier space 
where it becomes diagonal 


h 2 k 2 

F[Til>](k)=—F[il>](k). 

2m n 


(23.46) 


23.2.1.1 Pseudo-Spectral Methods 

The potential energy is diagonal in real space. Therefore, pseudo-spectral 
(Sect. 12.5.1) methods [281] use a Fast Fourier Transform algorithm (Sect. 7.3.2) 
to switch between real space and Fourier space. They calculate the action of the 
Hamiltonian on the wavefunction according to 
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Hip(x ) = V(x)ip(x) + T 1 


" frk 2 

. 2m p 


(k) 


(23.41) 


23.2.1.2 Finite Difference Methods 


In real space, the kinetic energy operator can be approximated by finite differences 
on a grid, like the simple 3-point expression (3.31) 


« + ,+<- 1 - 2« +0 2) 


2m p Ax 2 


(23.48) 


or higher order expressions (3.33) 

^ -r m+2 + i6c + i - 3oc + i^_ t - r m -i 

2m n 12 Ax 2 


■ 0(Ax a ) 


(23.49) 


l p 

h 2 l 

n p 


2m p Ax 2 (90 ^ m+3 20^ m+2+ 2^ m+1 18^ 

+ _ 20^'™- 2 + 90 ^“- 3 ) + 0(Ax ^ 


(23.50) 


etc. [282]. However, finite differences inherently lead to deviations of the dispersion 
relation from (23.46). Inserting t jj m = e ,k '" Ax we find 


E(k) = 


hr 2(1 — cos (kAx)) 


2m r 


np Ax~ 
for the 3-point expression (23.48), 

hr 15 — 16cos(£Ay) + cos(2kAx) 


(23.51) 


E(k) = 


2m, 


6Ax 2 


(23.52) 


for the 5-point expression (23.49) and 
h 2 


2 m, 


1 /49 3 1 \ 

- ( — — 3 cos(kAx) + — cos(2kAx) — — cos(3kAx) J (23.53) 


for the 7-point expression (23.50). Even the 7-point expression shows large deviations 
for k-values approaching k nax = r j Ax (Fig. 23.2). However, it has been shown that 
not very high orders are necessary to achieve the numerical accuracy of the pseudo- 
spectral Fourier method [283] and that finite difference methods may be even more 
efficient in certain applications [284]. 
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Fig. 23.2 (Dispersion of 
finite difference expressions) 
The dispersion relation of 
finite difference expressions 
of increasing order (23.48, 
23.49, 23.50 and the 
symmetric 9-point 
approximation [282]) are 
compared to the exact 
dispersion (23.46) of a free 
particle (dashed curve) 



23.2.2 Time Evolution 

A number of methods have been proposed [280, 285-287] to approximate the short 
time propagator (23.44). Unitarity is a desirable property since it guaranties stability 
and norm conservation even for large time steps. However, depending on the applica¬ 
tion, small deviations from unitarity may be acceptable in return for higher efficiency. 
The Crank-Nicolson (CN) method [288-290] is one of the first methods which have 
been applied to the time dependent Schroedinger equation. It is a unitary but implicit 
method and needs the inversion of a matrix which can become cumbersome in two 
or more dimensions or if high precision is required. Multistep methods [291, 292], 
especially second order [293] differencing (SOD) are explicit but only conditionally 
stable and put limits to the time interval At. Split operator methods (SPO) approxi¬ 
mate the propagator by a unitary product of operators [294-296] . They are explicit 
and easy to implement. The real-space split-operator method has been applied to 
more complex problems like a molecule in a laser field [297]. Polynomial approx¬ 
imations, especially the Chebishev expansion [298, 299], have very high accuracy 
and allow for large time steps, if the Hamiltonian is time independent. However, 
they do not provide intermediate results and need many applications of the Hamil¬ 
tonian. The short time iterative Lanczos (SIL) method [118, 300, 301] is very useful 
also for time dependent Hamiltonians. Even more sophisticated methods using finite 
elements and the discrete variable representation are presented for instance in [302, 
303]. In the following we discuss three methods (CN,SOD,SPO) which are easy to 
implement and well suited to solve the time dependent Schroedinger equation for a 
mass point moving in a one-dimensional potential. 
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23.2.2.1 Rational Approximation 

Taking the first terms of the Taylor expansion 


U(t n+U 4) = exp 


i At 

- H 

h 


i At 

= 1 - H 

h 


(23.54) 


corresponds to a simple explicit Euler step 


/ i At \ 

V’(Wt) = 11 — n~ H ) ^ tn -*• 


(23.55) 


From the real eigenvalues E of the Hamiltonian we find the eigenvalues of the explicit 
method 


i At 

A = 1- E 

h 


(23.56) 


which all have absolute values 


, At-E 2 

|A| = V 1 + ^> 1 - 

Hence the explicit method is not stable. 

Expansion of the inverse time evolution operator 


(23.57) 


U (4, t n+ 1 ) = U ( 4 + 1 , 4) 1 = exp 


i At 


-H 


= 1 


i At 


-H 


leads to the implicit method 
i At 

ip(t n+ 1) = ip(t n ) -— HV>(4+t) 

n 

which can be rearranged as 

/ i At \ _1 

4>(t n +i) = ( 1 + -j~ H J i/j(t„). 


(23.58) 


(23.59) 


Now all eigenvalues have absolute values < 1. This method is stable but the norm 
of the wave function is not conserved. Combination of implicit and explicit method 
gives a method [289, 290] similar to the Crank-Nicolson method for the diffusion 
equation (Sect. 21.2.3) 


/,. X , iAt rr (Wn+i) , 

“0(4+1) - v(t„) = —— H I— -h —— I - 


(23.60) 
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This equation can be solved for the new value of the wavefunction 

0(4+ t) = (l + i (l - i ^H\ 0(0 (23.61) 

which corresponds to a rational approximation 4 of the time evolution operator 
(Cayley’s form) 


EJ (C+1 5 C ) 


!+i tn H ' 


(23.62) 


The eigenvalues of (23.62) all have an absolute value of 


|A| 


/ EAt\ 1 / EAt\ V 1 

= |( 1 + ++) ('-‘-aJ-T 


4 hr 


= 1. 


, £ 2 4( 2 


(23.63) 


It is obviously a unitary operator and conserves the norm of the wavefunction since 

(23.64) 


= 1 


V + i§ H J V + ifh H ) V-i £k H )v+^ H ) 


as H is Hermitian = H and and (1 —i^ H ) are commuting operators. 


2h 

From the Taylor series we find the error order 
-l 


/ At \ 1 / At \ ( At At 2 7 \ / At \ 

(l + i — H ) ( 1 — i — H ) =( 1-i — H - T // 2 +••• 111 -i — H ) 

V 2h ) V 2 h ) \ 2 h Ah 2 )\ 2 h ) 

i At At 2 7 / i At \ 7 

= + ‘- = esp (-x") + 0( ^ ) - 

For practical application we rewrite [304] 

( 1+i l")' 

/ /It \ -1 / At \ (At \~ l 

= 11+ i— H) (-1 -i — H + 2\ = -l + 2( 1 +i — H) 

V 2h ) V 2d ) V 2 h ) 

hence 


(23.65) 


(23.66) 


( 1+ +Kf 


0(4+t) = 2(1+ i—£/ ) -0(4) - 0(4) = 2% - '0(C). 


(23.67) 


4 The Pade approximation (Sect. 2.4.1) of order [1, 1], 
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V’On+i) i s obtained in two steps. First we have to solve 

Then %i(t n+ \) is given by 
ip(t n+ i) = 2x - 


We use the finite difference method (Sect. 12.2) on the grid 
x m = mAx m — 0 • • • M Vm = V’fe, x m ) 


and approximate the second derivative by 

d 2 , C+i+C-i- 2 C , „, A 2^ 

g~2V>(tn,X m ) = -^- 1 ” )• 


Equation (23.68) then becomes a system of linear equations 


Xo " 


~r 0 ~ 

Xi 


r x 

_ Xm _ 


i 

.. 

i_ 


with a tridiagonal matrix 


, . hAt 

( 2 _1 
-1 2 

\ 

At 

/Vo 

Vi 

\ 

A = 1 — t-x- 

4m p Ax 1 

■■ -1 

+ 1 2h 





-1 2 ) 


\ 

Vm / 


The second step (23.69) becomes 




Xo 


r < i 


= 2 

Xi 

- 

ri 



_Xm _ 


i 

^ . 

i_ 


Inserting a plane wave 

tjj = e i( ** -UJ,) 


(23.68) 


(23.69) 


(23.70) 


(23.71) 


(23.72) 


(23.73) 


(23.74) 


(23.75) 
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Fig. 23.3 (Dispersion of the Crank-Nicolson method) The dispersion relation of the Crank- 
Nicolson method (23.95) deviates largely from the exact dispersion (23.98), even for small val¬ 
ues of the stability parameter a. The scaled frequency uAt/a is shown as a function of kAx/n 
for a = 0.1, 1, 2, 5, 10 (solid curves) and compared with the exact relation of a free particle 
uAt/a = ( kAx/ir) 2 (dashed cun’e) 


we obtain the dispersion relation (Fig. 23.3) 

2 h ( 2 kAx\ 2 

— tan(uMf/2) = -( — sin-) 

At 2m p \Z\jt 2 / 

which we rewrite as 


uiAt = 2arctan 


2 a , 

—r Sin 
7T” 


kAx 7r 
7T 2 


(23.77) 


with the dimensionless parameter 
7 T 2 HAt 

a = -—r. 

2m p Ax~ 


(23.78) 


For time independent potentials the accuracy of this method can be improved 
systematically [305] by using higher order finite differences for the spatial derivative 
(Sect. 23.2.1) and a higher order Pade approximation (Sect. 2.4.1) of order [M, M] 
for the exponential function 


e 


z 


n 


1-z/zT 

l+z/z^* 


+ 0(z 2M+l ) 


(23.79) 


to approximate the time evolution operator 
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exp 


/ • * v M 

(-x*)-n 


i-WH/mr + 0(W , r+1) . 


=1 1 + (iAt H/h)Iz. 


*(M) 


(23.80) 


However, the matrix inversion can become very time consuming in two or more 
dimensions. 


23.2.2.2 Second Order Differencing 

Explicit methods avoid the matrix inversion. The method of second order differencing 
[293] takes the difference of forward and backward step 


i) = Ut„)ip(t n ) 


(23.81) 


1p(t„+1 ) = U(t n +l, t n )l/)(t„) 
to obtain the explicit two-step algorithm 

ip(t„+ i) = ip(t n -1 ) + [U(t n+ 1 , t n ) - U~\t n , r„_!)] ip(t n ). 
The first terms of the Taylor series give the approximation 

V>(f„+i) = ip(t n -1 ) - 2 + 0({At ) 3 ) 
n 


(23.82) 


(23.83) 


(23.84) 


which can also be obtained from the second order approximation of the time derivative 
[306] 


. fc d il>(t + At) - ip(t - At) 

H ib = in—w = -. 

dt 2 At 

This two-step algorithm can be formulated as a discrete mapping 

/ ip(t n+ 1 )\ _ /-2 1 f-/7 l\ / ip(t n ) \ 

\ tp(t n ) /V 1 °/ VV’Oh-i)/ 

with eigenvalues 


(23.85) 


(23.86) 


H V h 2 

For sufficiently small time step [280] 

h 

At < - 


(23.87) 


max | E s | 


(23.88) 
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the square root is real, 


l^l 2 = 


E 2 At 2 

h 2 



E 2 At 2 \ 

~l9~ / 


= 1 


(23.89) 


and the method is conditionally stable and has the same error order as the Crank- 
Nicolson method (Sect. 23.2.2). Its big advantage is that it is an explicit method and 
does not involve matrix inversions. Generalization to higher order multistep differ¬ 
encing schemes is straightforward [291]. The method conserves [306] the quantities 
SH < ip(t + At)\ip(t) > and ill < ijj(t + At)\H\i/>(t ) > but is not strictly unitary 
[293]. Consider a pair of wavefunctions at times to and t\ which obey the exact time 
evolution 


ip(t\) = exp 



ip(to) 


(23.90) 


and apply (23.84) to obtain 


V'fe) = 


l At 

i At 

1 — 2- H exp 

n 

f^ H \. 


^ (to) 


(23.91) 


which can be written as 

(23.92) 


where the time evolution operator L obeys 



Expanding the sine function we find the deviation from unitarity [293] 

+ 2 (At \ 4 , 

C + C-l = -(—//) +••• = 0({At) A ) (23.93) 

which is of higher order than the error of the algorithm. Furthermore errors do not 
accumulate due to the stability of the algorithm (23.89). This also holds for deviations 
of the starting values from the condition (23.90). 

The algorithm (23.84) can be combined with the finite differences method 
(Sect. 23.2.1) 
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Fig. 23.4 (Dispersion of the Fourier method) The dispersion relation of the SOD-Fourier method 
(23.95) deviates from the exact dispersion (23.98) only for very high ^-values and approaches it for 
small values of the stability parameter a. The scaled frequency uiAt/a is shown as a function of 
kAx/n for a = 0.5, 0.75, 1 (solid curves ) and compared with the exact relation of a free particle 
uAt/a = ( kAx/n) 2 (dashed curve) 




,n +1 


„ 1 i At 

^ - 2 t- 


v m r m - 


h 2 


2m p Ax 2 


(C+1 + C-1 - 2C) 


(23.94) 


or with the pseudo-spectral Fourier method [306]. This combination needs two 
Fourier transformations for each step but it avoids the distortion of the dispersion 
relation inherent to the finite difference method. Inserting the plane wave (23.75) 
into (23.84) we find the dispersion relation (Fig. 23.4) for a free particle (V = 0): 


U) 


1 / HAt k~\ 1 / 

—arcsin I - I = — arcsinl a I -I I. 

At \ 2 nip ) At \ \ tt / J 


For a maximum /r-value 


7T 

k — _ 

''-max — 


Ax 


the stability condition (23.88) becomes 


x > At ^^lax 
— H 2m „ 


For small k the dispersion approximates the exact behavior 


hk 2 

2m p 


(23.95) 


(23.96) 


(23.97) 


to = 


(23.98) 
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Fig. 23.5 (Dispersion of the finite difference method) The dispersion relation of the SOD-FD 
method (23.99) deviates largely from the exact dispersion (23.98), even for small values of the 
stability parameter a. The scaled frequency uiAt/a is shown as a function of kAx/n for a = 
7 t 2 /4 ss 2.467, 1.85, 1.23, 0.2 (solid cur\>es) and compared with the exact relation of a free particle 
uAt/a = (kAx/n) 2 (dashed cun’e) 


The finite difference method (23.94), on the other hand, has the dispersion relation 
(Fig. 23.5) 


1 ( 4cr 2 (kAx\\ 

= — arcsin I —sin I -I I 


At 


V 2 // 


(23.99) 


and the stability limit 


At 

1 — _F — 

1 — , ^max — 

h 


2hAt 


m p Ax 2 7r 


4ct 
_ 2 ' 


(23.100) 


The deviation from (23.98) is significant for kAx/ir > 0.2 even for small values of 
a [306]. 


23.2.2.3 Split-Operator Methods 

The split-operator method approximates the exponential short time evolution oper¬ 
ator as a product of exponential operators which are easier tractable. Starting from 
the Zassenhaus formula [307] 

e X(A+B) = e AA e AB e A*C 2e ACi (23 . 101 ) 
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C 2 =\[B,A] C 3 = \[C 2 , A + 2B] ... (23.102) 

z o 

approximants of increasing order can be systematically constructed [295, 308] 
e XA+B) = e \A e \B + 0 ( X 2 } = £ XA £ XB £ X‘ 'C 2 + 0() 2 ) . . . _ (23.103) 

Since these approximants do not conserve time reversibility, often the symmetric 
expressions 


e HA+B) = e XAI2 e XB e XAI2 + Q ^ = £ XA/2 £ XB ,2 £ X^/A £ XB ,2 £ XA/2 + 

(23.104) 


are preferred. 

Split-Operator-Fourier Method 

Dividing the Hamiltonian into its kinetic and potential parts 


H = T + V = 


H 2 d 2 
2m P dx 2 


+ V(x) 


(23.105) 


the time evolution operator can be approximated by the time-symmetric expression 


1 At rp 

U{At) = e 2 >* r e 


i At 
h 


i At y 
2 h 1 


+ 0((At ) 3 ) 


(23.106) 


where the exponential of the kinetic energy operator can be easily applied in Fourier 
space [306, 309]. Combining several steps (23.106) to integrate over a longer time 
interval, consecutive operators can be combined to simplify the algorithm 

U{NAt) = U N {At) = e~^ T fe-TT^e-Tr^j (23.107) 

Real-Space Product Formulae 

Using the discretization (23.48) on a regular grid the time evolution operator 
becomes the exponential of a matrix 


/» 


-i At 


m p Ax 2 

h 

2m p Ax 2 


2m p Ax 2 

Yl + ft 

h ' nip Ax 2 


2m p Ax 2 


h Vm _ i _ h 

2mpAx 2 h ' mpAx 2 / 


U {At) = exp 
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ho + 2(3 -[3 \ 


exp 

—iAt 

P 7i+2/3 -(3 




( ~P 1m + 2(3 j 



with the abbreviations 


(23.108) 


lm ~ h Vm P ~2m P Ax 2 ’ 


(23.109) 


The matrix can be decomposed into the sum of two overlapping tridiagonal block 
matrices [294, 297] 5 


/ 70 + 2 13-13 \ 

~P + P 

\l2 + P ~P 

V -p '■/ 


/A, 


A 3 


Am-i / 


o 

o 


\ 


o ,71 + P 

-p 



-p 

\l2 + P 

0 

— 

V 

0 




\ 



(23.110) 


(23.111) 


The block structure simplifies the calculation of e lA,H « and e lA,H ‘ tremendously 
since effectively only the exponential functions of 2 x 2 matrices 


B m (t) = e 1tAi 


(23.112) 


have to be calculated and the approximation to the time evolution operator 

U(At) = e -i A ‘HJ2 e -iAtH ee -iAtHJ2 


M(f) \ 

Z 1 

\ 

(B i(f) 

\ 

5 3 (f) 

B 2 (At) 


BP f) 


v ■■■) 

\ 


l 

•/ 


(23.113) 


can be applied in real space without any Fourier transformation. To evaluate (23.112) 
the real symmetric matrix A m is diagonalized by an orthogonal transformation 
(Sect. 10.2) 


5 For simplicity only the case of even M is considered. 
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A = R~ l AR = R 


-i /At 0\ 

V 0 M) 


R 


and the exponential calculated from 


g -irA = | _ h - R -1 ar 


(—it) 2 , ~ , ~ 

-- -R-'ARR-'AR- 


= R- 1 


(—ir) 2 ~ 

1 -irA+ -- —AR H- R 


= R 


1 e- i ^R = R-'( e ~ 1TXl e _ iTX ^R. 


(23.114) 


(23.115) 


23.2.3 Example: Free Wave Packet Motion 

We simulate the free motion (V = 0) of a Gaussian wave packet along the x-axis 
(see Problem23.1). To simplify the numerical calculation we set h= 1 and m p = 1 
and solve the time dependent Schroedinger equation 


a , is 1 , 
I s,* = -2dT i ' 


(23.116) 


for initial values given by a Gaussian wave packet with constant momentum 


(-) 

\fl7T / 


1/4 


ipo(x) = ( — } e °*e 


ikox -x 2 /a 


(23.117) 


The exact solution can be easily found. Fourier transformation of (23.117) gives 
1 


- 1 r°° ■ , / a \V 4 r a 

4>k(t = 0) = -j= j dxe lkx ip 0 (x) = J exp |--(£ - & 0 ) J ■ 


(23.118) 


Time evolution in fc-space is rather simple 

• d -j k 2 - 

‘a* = y* 


(23.119) 


hence 


$k(,t) = e ' khl2 ^ k (t = 0) 


(23.120) 
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and Fourier back transformation gives the solution of the time dependent Schroedinger 
equation in real space 


ip(t, x ) = 


dk e lkx i/ji c (t) 




(2a \ 1/4 1 

(x-i^) 2 + a -f(a + 2it)~ 

\ 7T / a + lit P 

a + lit 


Finally, the probability density is given by a Gaussian 


\i>(t,x)\ 2 = 


1 


7T Va 2 + 4 1 2 


exp 


2 a 


- 4 r 2 


0 - *oO 


which moves with constant velocity ki, and kinetic energy 

L dx 0 (- t 10 ^ 0 = l (‘» + 0 ■ 


(23.121) 


(23.122) 


(23.123) 


Numerical examples are shown in Figs. 23.6, 23.7 and Table 23.1. 


23.3 Few-State Systems 


In the following we discuss simple models which reduce the wavefunction to the 
superposition of a few important states, for instance an initial and a final state which 
are coupled by a resonant interaction. We approximate the solution of the time depen¬ 
dent Schroedinger equation as a linear combination 


Fig. 23.6 (Conservation of 
norm and energy) The free 
motion of a Gaussian wave 
packet is simulated with the 
Crank-Nicolson method 
(CN), the second order 
differences method (SOD) 
with 3 point (23.48) 5 point 
(23.49) and 7-point (23.50) 
differences and with the 
real-space split-operator 
method (SPO). At = 10 -3 , 
Ar = 0.1, « = 1, jfc 0 = 3.77 
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Fig. 23.7 (Free wave-packet motion) The free motion of a Gaussian wave packet is simulated. The 
probability density is shown for the initial Gaussian wave packet and at later times t = 1, 2, 3, 4. 
Results from the second order differences method with 3 point differences (23.48, reel clash-dotted) 
and 5 point differences (23.49, blue dashed) are compared with the exact solution (23.122, thin 
black solid line). At = 10~ 3 , Ax = 0.1, a = 1, kg = 3.77 


Table 23.1 (Accuracy of finite differences methods) The relative error of the kinetic energy (23.123) 
is shown as calculated with different finite difference methods 


Method 

Ekin 

r pexacl 

E kin L kin 

E S C ‘ 

Crank-Nicolson (CN) with 3 point differences 

7.48608 

-1.6 x 10“ 2 

Second order differences with 3 point differences (SOD3) 

7.48646 

-1.6 x 10“ 2 

Second order differences with 5 point differences (SOD5) 

7.60296 

-4.6 x 10~ 4 

Second order differences with 7 point differences (SOD7) 

7.60638 

-0.9 x 10~ 5 

Split-operator method (SOP) with 3 point differences 

7.48610 

-1.6 x 10“ 2 

Exact 

7.60645 



M 

\m > (23 - 124) 

;=1 

of certain basis states \4>\ > • • • \4>m > 6 which are assumed to satisfy the necessary 
boundary conditions and to be orthonormalized 

< <j>i\(j)j >= Sij. (23.125) 


6 This basis is usually incomplete. 
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Applying the method of weighted residuals (Sect. 12.4) we minimize the residual 

|R >= ih^CjVMj > - > (23.126) 

j j 

by choosing the basis functions as weight functions (Sect. 12.4.4) and solving the 
system of ordinary differential equations 

0 - Rj =< (j>j\R >= i hCj - < <t>j\H\(f>j' > c y (23.127) 

j' 

which can be written 

M 

iHCi = ^ HtjCj (0 (23.128) 

j= i 

with the matrix elements of the Hamiltonian 


Hij =< 4>i | H | (j)j > . 

In matrix form (23.128) reads 


i h 


(23.129) 


/ Ci(t) ^ 


/ H hl ■ 




\H m ,i ■ 

■ H m ,m J 


Ci(0 \ 


(23.130) 


or more symbolically 

= HC(f). 


(23.131) 


If the Hamilton operator does not depend explicitly on time (H = const.) the formal 
solution of (23.131) is given by 


C = exp 


t 

— H 
ih 


C(0). 


(23.132) 


From the solution of the eigenvalue problem 

HC X = AC a (23.133) 

(eigenvalues A and corresponding eigenvectors Ca) we build the linear combination 
C = ’y^axC\e T ^ 1 . 


(23.134) 








540 


23 Simple Quantum Systems 


The amplitudes a\ can be calculated from the set of linear equations 

C(0) = ^a A C A . (23.135) 

A 

In the following we use the 4th order Runge-Kutta method to solve (23.131) numer¬ 
ically whereas the explicit solution (23.132) will be used to obtain approximate 
analytical results for special limiting cases. 

A time dependent Hamiltonian Hit) appears in semiclassical models which treat 
some of the slow degrees of freedom as classical quantities, for instance an electron 
in the Coulomb field of (slowly) moving nuclei 


Hit) = T + V ~ qj + V _ ijqi ' _ 

d y 47T£o|r - Rjit)\ y 47T£o|Rj(0 - R/'(0I 

or in a time dependent electromagnetic held 


(23.136) 


H(t) = T e , + V d -erW)- 


(23.137) 


23.3.1 Two-State System 


The two-state system (Fig. 23.8) (also known as two-level system or TLS) is the 
simplest model of interacting states and is very often used in physics, for instance in 
the context of quantum optics, quantum information, spintronics and quantum dots. 
Its interaction matrix is 


H = 



(23.138) 


and the equations of motion are 


ihC i — E\C\ -\- VC 2 

ihC 2 = E2C2 + VC\ 


(23.139) 


The interaction matrix can be diagonalized by an orthogonal transformation 
(Sect. 10.2) 


Fig. 23.8 Two-state system |2> 

model 


I1> 


V 


V 
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« = fiHfir = (\) 

with the rotation matrix 

cos ip — sin ip 
sin p cos ip 

The tangent of tp can be determined from (10.2) 



(23.140) 


(23.141) 


r = tan ip = —sign 



e 2 -e i 
2V 



(23.142) 


from which we find 


cos ip — 


1 

Vl + T 2 


sin ip 


■ 

Vl + T 2 


and the eigenvalues 


(23.143) 


A i = Ei-tV X 2 = E 2 + tV. 


(23.144) 


Finally the solution of (23.139) is given by (23.134) 




Ue 2 +tV)i 


For initial conditions 


Ci(0)=l C 2 (0) — 0 


solution of (23.135) provides the coefficients 

1 r 

A = 3-3 B = “3-3 

1 + T 2 1 + T 2 

and hence the explicit solution 



T+r 1 {^ El ~ TV)t 


^ek^ + rv )t \ 
- e r,i(E 2 +TV)t^ J 


(23.145) 


(23.146) 


(23.147) 


(23.148) 
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The occupation probability of the initial state is 

, 1 + T 4 2 T 2 / t\ 

c ' = (TT7¥ + (TTW cos ( (£2 - + 2TV) nJ ' (21149) 

It oscillates with the frequency 

HQ = JaV 2 + (E 2 - Ei) 2 (23.150) 

and reaches a minimum value 

n- T 2 \ 2 A£l (\ae\- jAE 2 + 4V 2 y AE 2 

Vl + T 2 / (4V 2 + AE 2 - \ AE\VAE 2 + 4V 2 ) 2 AE- + 4 V 2 

(23.151) 

Of special interest is the fully resonant limit. E\ = E 2 . Addition and subtraction of 
equations (23.139) here gives 

± C 2 ) = (E 1 ± V)(Ci ± C 2 ) (23.152) 

dr 

with the solution 

Cl ± C 2 = (C[(0) ± C 2 (0))e- i,(El±v)/n . (23.153) 

For initial conditions given by (23.146) the explicit solution is 

,, Fr , , Vr 1 + cos ^ 

Ci = e- ,tEl/H cos — | Ci | 2 = cos 2 —— = - ^ (23.154) 

h n 2 




C 2 — -ie 


Vt 


sin — \C 2 \ = sin" — = 


Vt 1 — cos 


(23.155) 


h h 2 

At resonance the two-state system oscillates between the two states with the period 


irh 

T = —. 
V 


(23.156) 


Numerical results are shown in Fig. 23.9. 
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(a) AE=0 V=2 (c) AE=1 V=1 



Fig. 23.9 (Numerical simulation of a two-state system) The equations of motion of the two-state 
system (23.139) are integrated with the 4th order Runge-Kutta method. For two resonant states the 
occupation probability of the initial state shows oscillations with the period (23.156) proportional 
to V~ l . With increasing energy gap Ei — E\ the amplitude of the oscillations decreases 


Fig. 23.10 Two-state 
system in an oscillating field 




I1> 


23.3.2 Two-State System with Time Dependent Perturbation 


Consider now a 2-state system with an oscillating perturbation (Fig. 23.10) (for 
instance an atom or molecule in a laser field) 


H = 


( E, V(f)\ 
\V(t) e 2 ) 


V(t) = Vo COS Lot. 


(23.157) 


The equations of motion are 


iftC, = £iCi + V(t)C 2 
ihC 2 = V(t)C\ + E 2 C 2 ' 


(23.158) 
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After the substitutions 

El, 

Ci = e ih u i 

n Ei, 

C 2 — C U 2 

£2 - £1 

W21 = -r- 

n 

they become 


(23.159) 

(23.160) 


iMi = V(f)e a ^ L, u 2 = f ( e _i(W2I_w)t + e - i ^+^') n 2 j 

ifii<2 = y(t)e^s SLf «i = f (e i(W2 >-^ r + e i( “ 21+w)r ) m. 

At larger times the system oscillates between the two states. 7 Applying the rotating 
wave approximation for u> ~ u> 2 \ we neglect the fast oscillating perturbation 


i M! = —e“ i(W21 - w)r n 2 
2 


iM 2 = —e^-^'m 
2 

and substitute 


u\ = a\ 


e -i(W21—0 j)1 


(23.162) 

(23.163) 


(23.164) 


to have 

i^(«i - aii(w 2 i - w))e- i(W2 ‘- w)? = y e- i(W21 - w)t n 2 


i hiii = —e 
2 


^0 -i(W21-W)( -i(W21-W)f 


«1 


or 


Vo 

iha\ = h{u> - LU 2 l)fll + -y W 2 


... Vo 

lto 2 = yfll 


(23.165) 

(23.166) 


(23.167) 

(23.168) 


7 So called Rabi oscillations. 
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which shows that the system behaves approximately like a two-state system with 
a constant interaction Vq/2 and an energy gap h(t 02 i — to) — E 2 — E\ — hui 
(a comparison with a full numerical calculation is shown in Fig. 23.1 1). 


23.3.3 Superexchange Model 


The concept of superexchange was originally formulated for magnetic interactions 
[310] and later introduced to electron transfer theory [311]. It describes an indirect 
interaction through high energy intermediates (Fig. 23. 12). In the simplest case, we 
have to consider two isoenergetic states i and / which do not interact directly but 
via coupling to an intermediate state v. 

The interaction matrix is 


V=1 Ato=0 V=0.5, 1.0 Aco=5.0 




Fig. 23.11 (Simulation of a two-state system in an oscillating field) The equations of motion 
(23.158) are integrated with the 4th order Runge-Kutta method. At resonance the system oscillates 
between the two states with the frequency V/h. The dashed cun’es show the corresponding solution 
of a two-state system with constant coupling (Sect. 23.3.1) 
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Fig. 23.12 Superexchange 
model 
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0 Vi 0 

H = | V l E 2 V 2 

0 v 2 o 


(23.169) 


For simplification we choose V\ — V 2 . 

Let us first consider the special case of a resonant intermediate state E 2 = 0: 

0 V 0\ 

H = | V 0 V ) . (23.170) 

0 V 0/ 

Obviously one eigenvalue is Ai =0 and the corresponding eigenvector is 


C, = . (23.171) 



The two remaining eigenvalues are solutions of 
-A V 0 

V -A V | = A(—A 2 + 2V 2 ) (23.172) 

0 V -A 


0 = det 


which gives 


A2,3 = ±s/2V. 


The eigenvectors are 



From the initial values 


( o\ + a 2 + 03 \ / 1 \ 

V2 a 2 - V2a 3 ) = ( ° 

—+ a 2 + a 2 J \0j 


the amplitudes are calculated as 


Cl\ = — di = U'X — — 

2 4 


(23.173) 


(23.174) 


(23.175) 


(23.176) 


and finally the solution is 
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c = 


1 

2 




i -,-Jivt 


1 

4 



fisin^r 

V-t + tc„ sS ® 



(23.177) 


Let us now consider the case of a distant intermediate state V £ 2 1 - A] =0 and 
the corresponding eigenvector still provide one solution. The two other eigenvalues 
are approximately given by 


^ 2,3 



+ 2 V 2 + 


Ei 

2 


Ei Ei 
— ± —(1 ■ 
2 2 



A 2 


Ei + 


2V 2 
~E 2 


A 3 


2V 2 

~e7 


and the eigenvectors by 


C 


2 ~ 




From the initial values 

( 1 \ / At + 02 + <23 \ 

0 I = | ciiXi + (23A3 J 

0 J y —a\ + Cli + fl3 J 

we calculate the amplitudes 



and finally the solution 


/ I(l+e-Hi^) \ 

2L e k E * - 

£2 e £ 2 e 

, 1 2V 2 , 

V i(-l + e '*'**) / 


(23.178) 

(23.179) 


(23.180) 


(23.181) 


(23.182) 


C 


(23.183) 
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Fig. 23.13 (Numerical simulation of the superexchange model) The equations of motion for the 
model (23.169) are solved numerically with the 4th order Runge-Kutta method. The energy gap 
is varied to study the transition from the simple oscillation with uj = \/2V/h (23.177) to the 
effective two-state system withu) = 14//-/?i(23.184). Parameters are V\ = V 2 = 1, E\ = E 3 = 0, 
£2 = 0, 1,5,20. The occupation probability of the initial (solid curves), virtual intermediate 
(dashed cun’es) and final ( dash-dotted curves) state are shown 


The occupation probability of the initial state is 

|C!| 2 = -|l + e <STT'| 2 = cosM — t) (23.184) 

which shows that the system behaves like a 2-state system with an effective interaction 
of 


V 2 

Veff = —. (23.185) 

£ 2 

Numerical results are shown in Fig. 23.13. 


23.3.4 Ladder Model for Exponential Decay 

For time independent Hamiltonian the solution (23 .1 32) of the Schroedinger equation 
is a sum of oscillating terms and the quantum recurrence theorem [312] states that the 
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Fig. 23.14 Ladder model 


li>- 


lf,n> 


system returns to the initial state arbitrarily closely after a certain time T r . However, 
if the initial state is coupled to a larger number of final states, the recurrence time 
can become very long and an exponential decay observed over a large period. The 
ladder model [313, 314] considers an initial state |0 > interacting with a manifold 
of states 11 > ... | n >, which do not interact with each other and are equally spaced 
(Fig. 23. 14) 


/0 V ■■■ v\ 

V E 1 


H = 


Ej = E, + (j - 1 )AE. 


(23.186) 


V V E n ) 


The equations of motion are 


n 


iftCo = vY J Cj 


i hCj = EjCj + VCq. 


(23.187) 


For the special case AE = Owe simply have 



(23.188) 


with an oscillating solution 



(23.189) 


Here the n states act like one state with an effective coupling of V *Jn. 
For the general case AE / Owe substitute 
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and have 

i Mj&k 1 = VC 0 . (23.191) 

Integration gives 

Uj = — f e“« f ' C 0 (t')dt' (23.192) 

Jto 

and therefore 

Cj = ^r [ e'i u '- t) Co(t , )dt'. (23.193) 

'h J , 0 

With the definition 

Ej — j * YiAlo (23.194) 

we have 

Co = = -f=-Y / e 9«'-« c (23.195) 

We replace the sum by an integral over the continuous variable 

to — j Auj (23.196) 


and extend the integration range to — oo • • • oo. Then the sum becomes approximately 
a delta function 


j =-00 


jAu{t'—t) 


.'V 


ZW Auj 


and the final result is an exponential decay law 
2ttV 2 2ttV 2 

Co = _ ^ Co = _ ^r p(£)Co 

with the density of final states 


P(E) = 


1 

HAuj 


1 

A E' 


(23.197) 


(23.198) 


(23.199) 


Numerical results are shown in Fig. 23.15. 
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Fig. 23.15 (Numerical 
solution of the ladder model) 
The time evolution of the 
ladder model (23.187) is 
calculated with the 4th order 
Runge-Kutta method for N 
= 50 states and different 
values of the coupling V 



23.3.5 Semiclassical Curve Crossing 


In the following we study simple models for the transition between two electronic 
states along a nuclear coordinate Q ‘ Within the crude diabatic model the wavefunc- 
tion takes the form 


V = 


(xi(Q,0 

WQ,t) 


) 


(23.200) 


where the two components refer to the two electronic states. 

The nuclear wavefunctions xi ,2 obey a system of coupled equations (M is the 
reduced mass corresponding to the nuclear coordinate) 


ih'I' = MV 


^ d 2 , (ExiQ) f(©\] 

2M8Q 2 \V(Q) E 2 {Q)J\ ■ 


(23.201) 


Here E\ 2 {Q) are the diabatic potential energy surfaces which cross at a point Q c 
and V(Q) is the coupling matrix element in the diabatic basis. 

According to Ehrenfest’s theorem, the average position 

G(0 = J [lxi(e,0| 2 + IX2(e, Ol 2 ] QdQ (23.202) 


'For a diatomic molecule, e.g. the nuclear coordinate is simply the distance R of the two nuclei. 
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obeys an equation of motion which looks very similar to its classical counterpart 

d 2 — — d 

M—Q(t) = F = - V eff 

df- ’ dQ n 


= J d Q (xi(G.0*X2(G,0*)^ 


~m E 

-£>V(Q) 


o d Q V(Q) \ / Xl (g, t)\ 

--SQE 2 (Q)J\X2(Q,t ))' 

(23.203) 


The semiclassical approach approximates both nuclear wavefunctions as one and 
the same narrow wave packet centered at the classical position Q(t) = Q(t). Equa¬ 
tion (23.203) then becomes 




/£t(G(0) V{Q(t)) \ (a{t)\ 
V V(Q(t)) E 2 (Q(t)) J \b(t) J 


Substitution of 


Xi(2. t) = Xz(Q, t ) = HQ, t ) 


(23.204) 


in (23.200) 


>F = 


(a(t)\ 


HQ, t) 


(23.205) 


and taking the average over Q, which in fact means to replace Q by Q(t), the 
semiclassical approximation is obtained: 


(a(t)\ ( Ei 


(<2(0) V(G(0) 

V(Q(t)) E 2 (Q(t)) 


)(«: 0- 


(23.206) 


In Problem 23.5 we compare the solutions of (23.201) and (23.206). The two wave 
packets are propagated with the split-operator-Fourier transform method Sect. 23.2.2. 
For a small time step At the propagator is approximated as a product 


exp 


At 

~Ih H 


— exp 


n d 2 

lAt 4MdQ 2 


exp 


\—(E\(Q) V(G)\ 
{ i h V V(Q) E 2 (Q)) 


exp 


iAt 


h d 2 
4M8Q 2 


(23.207) 
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where the kinetic energy part is evaluated in Fourier space and the potential energy 
part requires diagonalization of a 2 x 2 matrix for each grid point. From the resulting 
wavefunction the average position Q(t) is calculated which is needed to define the 
trajectory for the semiclassical approximation. Equation 23.206 is then solved with 
the Runge-Kutta method. The initial wavefunction is a Gaussian wave packet on one 
of the diabatic surfaces with constant momentum (as in 23.1 17). Figure 23.16 shows 
an example from Problem 23.5. 


23.3.6 Landau-Zener Model 

This model describes crossing of two states, for instance for colliding atoms or 
molecules [315, 316]. It is assumed that in the vicinity of the crossing point the 
interaction V is constant and the time dependency of the energy gap is linearized 
(Fig. 23.17) 

V(Q{t)) = V (23.208) 

AE(t) = E 2 {Q{t )) - Ei(Q(t)) = AE 0 + vt. (23.209) 

The Hamiltonian matrix of the Landau-Zener model is 



time 


Fig. 23.16 (Semiclassical approximation of a curve Crossing) The crossing between two states is 
simulated for coupling V = 1.23, velocity = 12 and slope = 0.4. (Problem23.5). Top the semiclas¬ 
sical approximation (black) reproduces the occupation probability from the full quantum calculation 
(red) quite accurately. Generally, it shows more pronounced oscillations than the quantum calcu¬ 
lation with wave packets of finite width. Bottom If the initial velocity is large enough, acceleration 
is not important and the classical position (black) as well as the energy gap (red) become linear 
functions of time 
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= (° V ) 
\ V AE(t) ) 


(23.210) 


For small interaction V or large velocity ^A E = Q-^AE the transition probability 
can be calculated with perturbation theory to give 


2ttV 2 


(23.211) 


This expression becomes invalid for small velocities. Here the system stays on the 
adiabatic potential surface, i.e. P —»• 1. Landau and Zener found the following 
expression which is valid in both limits: 


Plz = 1 - exp 



(23.212) 


In case of collisions multiple crossing of the interaction region has to be taken into 
account (Fig. 23. 18). 

Numerical results from Problem23.6 are shown in Fig. 23.19. 



Fig. 23.17 Slow atomic collision 



^-r = P i C —'z) +C — P 1 ) P 2= 2P C- P > 


Fig. 23.18 Multiple passage of the interaction region 
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Fig. 23.19 (Numerical solution of the Landau-Zener model) Numerical calculations (solid curves) 
are compared with the Landau-Zener probability (23.212, dashed lines) and the approximation 
(23.211, dotted lines) The velocity is dAE /dt = 1. (Problem23.6) 


23.4 The Dissipative Two-State System 

A two-state quantum system coupled to a thermal bath serves as a model for magnetic 
resonance phenomena, coherent optical excitations [317, 318] and, quite recently, 
for a Qubit, the basic element of a future quantum computer [319, 320]. Its quantum 
state can not be described by a single wavefunction. Instead mixed quantum states 
have to be considered which can be conveniently described within the density matrix 
formalism [277]. 


23.4.1 Equations of Motion for a Two-State System 

The equations of motion for a two-state system are 


ihpn = H\2P2\ — P12H21 


(23.213) 


ihf>22 = 7/21P12 — P2\H\2 


(23.214) 
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ihpn = (H u — H 22 )pn + H 1 2 (p 22 — Pit) (23.215) 

—ihp 2 \ — (H n — H 22 )p 2 i + H 2 i(p 2 i ~ Pit) (23.216) 


which can be arranged as a system of linear equations 9 


/ Pit \ 


r-t 

tR 

£ 

1 

0 

0 


/ Pll\ 

P 22 


rf 

ER 

1 

£ 

0 

0 


P22 

P 12 


— H\2 H\2 H 11 — H22 0 


Pl2 

\ P 21 / 


tR 

1 

<N 

ac 

O 

a? 

1 


\ P21 / 


(23.217) 


23.4.2 The Vector Model 

The density matrix is Hermitian 

Pij = P*i (23.218) 

its diagonal elements are real valued and due to conservation of probability 

Pit + P 22 = const. (23.219) 

Therefore the four elements of the density matrix can be specified by three real 
parameters, which are usually chosen as 

x = 23tp2i (23.220) 

y = 23/921 (23.221) 

Z = pit P 22 (23.222) 

and satisfy the equations 

j-291(p 21 ) = ((Hu - H 22 )2%(p2i) + 23(H n )(pn - p 22 )) (23.223) 

dr n 

^23(p2i) = l ((H u - H 22 )2m P 2i) - 2fH(H n )(pn - P 22 )) (23.224) 

dr h 

^(Pit - P 22 ) = \ (3(ff 12 )2SR(p 21 ) + SRffi 2 23(p2i)). (23.225) 

dr h 


9 The matrix of this system corresponds to the Liouville operator. 









23.4 The Dissipative Two-State System 


557 


Together they form the Bloch vector 


r = 


(23.226) 


which is often used to visualize the time evolution of a two-state system [321], In 
terms of the Bloch vector the density matrix is given by 


(“MITH--* 


x+iy 1 —z / o ' 

2 2 ' z 


(23.227) 


with the Pauli matrices 


-(?;)• *.-e.,)• 


(23.228) 


From (23.223-23.225) we obtain the equation of motion 



- _ v H n -H 12 _ - 23(Hi 2 ) 
■ h *■ 


H\ i H 2 2 _ 

A h 1 

v 23(tfi 2 ) 


II \ 
2iH(tfi 2 ) 
h 

2!H(Hi 2 ) 


h + y n 

which can be written as a cross product 


(23.229) 


—r = uxr 

dr 


(23.230) 


with 


~pH 12 

UH n -H 2 i) j 


(23.231) 


Any normalized pure quantum state of the two-state system can be written as [322] 


\i> >= 




(23.232) 


corresponding to the density matrix 




COS 


2 e 


e _i< ^ sin f cos § 


e 1 ^ sin | cos | 


sin 


2 e 


) 


(23.233) 
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Fig. 23.20 (Bloch sphere) Left Any pure quantum state of a two-state system can be represented 
by a point on the Bloch sphere. Right The poles represent the basis states. Mixed quantum states 
correspond to the interior of the sphere, the central point represents the fully mixed state 


The Bloch vector 


r = 


^cos0sin6^ 
sin <fi sin 6 
cos 6 


(23.234) 


represents a point on the unit sphere (the Bloch sphere, Fig. 23.20). Mixed states 


correspond to the interior of the Bloch sphere with the fully mixed state p 
represented by the center of the sphere (Fig. 23.20). 



23.4.3 The Spin - 1 / 2 System 


An important example of a two-state system is a particle with spin ^. Its quantum 
state can be described by a two-component vector 

(SHOHO) 


(23.235) 


where the two unit vectors are eigenvectors of the spin component in z-direction 


corresponding to the eigenvalues s z 
given by the Pauli matrices 


h 

Si = 2 ai 


± 4. The components of the spin operator are 


(23.236) 


and have expectation values 
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( V / CfCa+CICi \ 

2to-»( a 4 ja 

cj z ) ^ ' y ici| 2 -ic 2 | 2 ■ 


(23.237) 


The ensemble average for a system of spin-4 particles is given by the Bloch vector 


' P21+P12 ' 

<S> = h[ \ = Jr. 

Pn~P 22 I ^ 


(23.238) 


The Hamiltonian of a spin -\ particle in a magnetic field B is 


H = — 7 —crB = 
2 


h( B z B x -iB y \ 
2 \ B x + i By — B- J 


(23.239) 


from which the following relations are obtained 
2 


7 B x = --fRH 12 
h 

(23.240) 

7 B y = 

(23.241) 

_ H n - H 22 

7 B z = t 

a 

(23.242) 

UJ = — 7 B. 

(23.243) 

The average magnetization 


h 

m = 7 < S > = 7 — r 

2 

(23.244) 

obeys the equation of motion 


d 

— m = — 7 B x m. 
dr 1 

(23.245) 


23.4.4 Relaxation Processes - The Bloch Equations 

Relaxation of the nuclear magnetization due to interaction with the environment 
was first described phenomenologically by Bloch in 1946 [323]. A more rigorous 
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description was given later [324, 325] and also applied to optical transitions [326]. 
Recently electron spin relaxation has attracted much interest in the new field of 
spintronics [327] and the dissipative two-state system has been used to describe the 
decoherence of a Qubit [328]. 


23.4.4.1 Phenomenological Description 


In thermal equilibrium the density matrix is given by a canonical distribution 


tr( e -/3«) 


(23.246) 


which for a two-state system without perturbation 


Ho = 

( f -f) 


(23.247) 

becomes 

/ e-W 2 

\ 


p eq = 

I e PA/2 +e -flA/2 

& HA/2 ) 

(23.248) 


V 

e /M/2 +e -/?4l/2 J 


where, as 

usually (3 = 

l/lcgT. If the energy gap is 

very large A ksT like for an 


optical excitation, the equilibrium state is the state with lower energy 10 


P 


eq — 



(23.249) 


The phenomenological model assumes that deviations of the occupation difference 
from its equilibrium value 


P Z- ™h(^) 


(23.250) 


decay exponentially with a time constant 7) (for NMR this is the spin-lattice relax¬ 
ation time) 


— (Pit - P22 ) = - — [(Pit - P22) ~ (Pit 

dt\Rel /1 


m ■ 


(23.251) 


10 We assume A > 0, such that the equilibrium value of z = pu — p 22 is negative. Eventually, the 
two states have to be exchanged. 
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The coherence of the two states decays exponentially with a time constant 73 
which is closely related to 7) in certain cases" but can be much smaller than 7i if 
there are additional dephasing mechanisms. The equation 


dt | Rel 


Pn = 


T 2 


Pn 


(23.252) 


describes the decay of the transversal polarization due to spatial and temporal differ¬ 
ences of different spins (spin-spin relaxation), whereas for an optical excitation or a 
Qubit it describes the loss of coherence of a single two-state system due to interaction 
with its environment. 

The combination of (23.245) and the relaxation terms (23.251, 23.252) gives 
the Bloch equations [323] which were originally formulated to describe the time 
evolution of the macroscopic polarization 


dm 

dr 


= — 7 B x m — R(m — m f9 ) 



For the components of the Bloch vector they read explicitly 


(23.253) 


d 

dr 



/ 

-1/72 

-^(77n - H 22 ) 

-*9*i 2 \ 

/ x\ 


i(77n - H 22 ) 

-1/73 

-|9i77, 2 


V 

*3*12 

S«*!2 

—1/7) ) 

\ z J 


( : ) 

\W TiJ 

(23.254) 


23.4.5 The Driven Two-State System 


The Hamiltonian of a two-state system (for instance an atom or molecule) in an 
oscillating electric field Ee,~ lulfl with energy splitting A and transition dipole moment 
p is 


-(■ 


4 -pEeT^f* 
-pEe iUft -f 


) 


The corresponding magnetic field 


B x = —pE cosw ft 

7ri 


(23.255) 


(23.256) 


"For instance 73 = 27i for pure radiative damping. 
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By 


2 

— /i£ sinu;/-r 

7 h 


A 


(23.257) 


B z =-- (23.258) 

7 n 

is that of a typical NMR experiment with a constant component along the z-axis and 
a rotating component in the xy— plane. 


23.4.5.1 Free Precession 


Consider the special case B z = const, 
tonian matrix is diagonal 



with the Larmor-frequency 


B x = B Y = 0. The corresponding Hamil- 

(23.259) 


£ 2 0 = — = -7#o. 
a 

The equations of motion for the density matrix are 
d , , (Pit - P22) ~ (Pn - P22) 

a'"" " ni) =- f, - 


d 1 

1 ^7rPl2 = fiA)Pl 2 - lb— Pn 
at T 2 


with the solution 


(23.260) 


(23.261) 

(23.262) 


(pn - P 22 ) = (pH - P 22 ) + [(Pn(0) - P 22 (0)) - (pH - P 22 )]e“ f/7 ' 1 (23.263) 


Pn = Pnm~ iS2ot ~‘ /Tl - (23.264) 

The Bloch vector 

( (xo cos P2ut — yo sin t)e~'^ T2 \ 

(y 0 cos f2 0 r + xq sin ^2 0 t)e~^ Tl J (23.265) 

+ (zo - z eq )e~ ,/T ' ) 

is subject to damped precession around the z—axis with the Larmor frequency 
(Fig. 23.21). 
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Fig. 23.21 (Free precession) The Bloch equations (23.254) are numerically solved with the 4th 
order Runge Kutta method. After excitation with a short resonant pulse the free precession is 
observed. Left The occupation difference z = pit — p 22 decays exponentially to its equilibrium 
value. Right In the jcy-plane the Bloch vector moves on a spiral towards the equilibrium position 
(x = 0, y = 0) 


23.4.5.2 Stationary Solution for Monochromatic Excitation 

For the two-state system (23.255) with 

H n - H 22 = A = hC2 0 (23.266) 


H 12 = Vo (cos u> ft — isincu/f) (23.267) 

the solution of the Bloch equations (23.253) 

/ —1/72 ~&0 ^ sinw / f ^ 

£2 0 1/7*2 -^co suj/t 

y-^sinw/f ^cosujft —l/Ti y 

(23.268) 




can be found explicitly [317]. We transform to a coordinate system which rotates 
around the z-axis (Sect. 14.3 on page 330) with angular velocity utf 


cos (coft) sin(ujft) 0 
— sin(w/f) cos (w/t) 0 
0 0 1 



= A(t) 


y' 

z' 


X 

y 

z 


X 

y 

z 


(23.269) 
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Then 


d 

dt 





( AA- 1 


AKA' 1 ) 




(23.270) 


with 

/ -l/r 2 ~Q 0 ^sin ojft \ 

^=1 ^0 -1 /T 2 -^cos LOft 

y — ^ sinw/t ^ cosujft —l/Ti y 

The matrix products are 


(23.271) 
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II 

II 

1 

e 

0 0 

AKA" 1 = ( 

-i/t 2 

1 

1 

% 

V 0 

0 0/ 

V 0 

2 Vo 
h 

-1/7)/ 


(23.272) 


and the equation of motion simplifies to 


( 


x' 

y 

z' 


) 


I Uf T?0 

^o-UJf -jr 

V 0 ^ 


0 \ 


2Vo 

h 

l 


7i / 



(23.273) 


For times short compared to the relaxation times the solution is approximately given 
by harmonic oscillations. The generalized Rabi frequency Q r follows from [329] 


i£2rx' — (ujf — )y' 


(23.274) 


i tiny' = (^2 q — u>f)x' — 



(23.275) 


i 

h 


as 


(23.276) 


C2r = 


J(^o-U)f ) 2 + (^) ■ 


(23.277) 
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Fig. 23.22 (Monochromatic Excitation) The Bloch equations are solved numerically with the 4th 
order Runge-Kutta method for a monochromatic perturbation with u> = 4, Vo = 0.5. Parameters 
of the two-state system are ojq = 5, z e q = —1.0 and 7i = 73 = 5.0. The occupation difference 
z = pi i — p 22 initially shows Rabi oscillations which disappear at larger times where the stationary 
value z = —0.51 is reached 


At larger times these oscillations are damped and the stationary solution is approached 
(Fig. 23.22) which is given by 


1+ 4-17)73+ 7f( W/ - 

The occupation difference 


27 2 2 f(f2 0 -w/) 

-273 f 
^o) 2 \l + T$(u f - C2 0 ) 2 


Z = p 11 — P22 


= Z "( 1_ T 


4^-r 1 r 2 


+ 4%T l T 2 + T?(u f -!2 0 y 


) 


(23.278) 


(23.279) 


has the form of a Lorentzian. The line width increases for higher intensities (power 
broadening) 


73 


V 2 

—J 1+4-5- T {T 2 


h 2 


and the maximum 


z(£2p) = 1 

1 + 4 ^ 7 ’ 1 7’ 2 


(23.280) 


(23.281) 


approaches zero (saturation) (Figs. 23.23, 23.24). 
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Fig. 23.23 (Resonance line) The equations of motion of the two-state system including relaxation 
terms are integrated with the 4th order Runge-Kutta method until a steady state is reached. Para¬ 
meters are ljq = 5, z e q = —0.8, V = 0.01 and T\ = T 2 = 3.0. 6.9. The change of the occupation 
difference is shown as a function of frequency ( circles ) and compared with the steady state solution 
(23.278) 


Fig. 23.24 (Power 
saturation and broadening) 
The resonance line is 
investigated as a function of 
the coupling strength V and 
compared with the stationary 
solution (23.278) to observe 
the broadening of the line 
width (23.280). Parameters 
are uo = 5 ,z eq = —1.0, 

Ti = T 2 = 100 and 
V = 0.5, 0.25, 0.125, 
0.0625,0.03125 



frequency © 


23.4.5.3 Excitation by a Resonant Pulse 

For a resonant pulse with real valued envelope Vq( 0 and initial phase angle <Pq 


H n = Vo(t)e~ i(i2o,+s>o) 


the equation of motion in the rotating system is 
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-^Fsm0 o \ 

-^cos 0 O 

~k ) 


/ ) + | 0 

T\ 


(23.282) 


If the relaxation times are large compared to the pulse duration this describes approx¬ 
imately a rotation around an axis in the xy —plane (compare with 14.24) 


d , , 2 Vo 0) 

-r' « W(t)r' = -^—Wor' 
dr n 


(23.283) 


0 0 — sin 0o 

Wo = | 0 0 — cos 0o 

^ sin 0o cos 0o 0 

Since the axis is time independent, a formal solution is given by 

r'(f) = e w & ^ df 'r'(0) = e Wo<!,(0 r'(0) 
with the phase angle 

r‘ 2Vo(f) , 

0(t) = / ov dt'. 


(23.284) 


(23.285) 


(23.286) 


Now, since 


^o = 


— sin 2 0q — sin 0q cos 0q 0 
sin 0q cos 0o — cos 2 0q 0 
0 0-1 


(23.287) 


Wo 3 = - Wo 


(23.288) 


W 0 4 = -W 2 


(23.289) 


the Taylor series of the exponential function in (23.285) can be summed up 

e w ° 0 = 1 + 0W O + l ~0 2 W°o + ^<? 3 W 0 3 + • • • 


7 (0 2 0 4 
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Fig. 23.25 (Rotation of the 


z’ 


Bloch vector by a resonant 
pulse) A resonant pulse 
rotates the Bloch vector by 
the angle <f> around an axis 
in the x'y'-plane 



= 1 + Wq (1 — cos 0) + Wq sin 0 

( 1 — sin 2 0q (1 — cos 0) — sin 0 O cos <J 0 (1 — cos 0) — sin 0q sin 0 

— sin 0o cos 0o (1 — cos 0) 1 — cos 2 0q (1 — cos 0) — cos 0o sin 0 

sin 0q sin 0 cos 0q sin 0 cos 0 

( cos 0q sin 0o 0 \ / 1 0 0 \ / cos 0q — sin 0 O 0 \ 

— sin 0q cos 0o 0 | ( 0 cos 0 — sin 0 1 I sin 0q cos 0 o 0 | . 

0 0 1 J y 0 sin 0 cos 0 J y 0 0 1 J 


(23.290) 


The result is a rotation about the angle 0 around an axis in the xy— plane deter¬ 
mined by 0q (Fig. 23.25), especially around the x— axis for 0o = 0 and around the 
y —axis for 0 O = f. 

After a n —pulse (0 = i r) the "-component changes its sign 

/ cos(2<J>o) — sin(20o) 0 \ 

r'= j - sin(20 o ) - cos(2^ 0 ) 0 J r(0). (23.291) 

V o o -\) 

The transition between the two basis states z = — 1 and z — 1 corresponds to a spin 
flip (Fig. 23.26). On the other hand, a 7r / 2 ~P L1 l se transforms the basis states into a 
coherent mixture 



r 


(23.292) 
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Fig. 23.26 (Spin flip by a 7r-pulse) The equations of motion of the Bloch vector (23.268) are 
solved with the 4th order Runge-Kutta method for an interaction pulse with a Gaussian shape. 
The pulse is adjusted to obtain a spin flip. The influence of dephasing processes is studied. 
7j = 1000, tp = 1.8, Vo = 0.25. The occupation difference p\\ — p 22 = z (solid curves) and 
the coherence |pi 2 1 = \ + y 2 (broken curves) are shown for several values of the dephasing 

time T 2 = 5, 10, 100, 1000 


23.4.6 Elementary Qubit Manipulation 

Whereas a classical bit can be only in one of two states 



the state of a Qubit is a quantum mechanical superposition 


(23.293) 




(23.294) 


The time evolution of the Qubit is described by a unitary transformation 

|>-» U\ip > (23.295) 

which is represented by a complex 2x2 unitary matrix that has the general form 
(see also Sect. 14.15) 

U =(- e Zp*J a *) \<*\ 2 + \P\ 2 = l, det U = e' v . 


(23.296) 
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The Bloch vector is transformed with an orthogonal matrix A, which can be found 
from (23.227) and the transformed density matrix UpU~ l 

1 91 ((a 2 - f3 2 )e~ ilp ) 3 ((ct 2 + /3 2 )e~^) -291 (ape~^) \ 
r Ar A = [ 3 ((/3 2 - a 2 )&~' v ) 91 ((ct 2 + /3 2 )e~ iv ) 23 (a/ie"^) ) . 

V 291 (a*p) 2 3(a*/?) (|a|‘- |/3| 2 ) / 

(23.297) 

Any single Qubit transformation can be realized as a sequence of rotations around 
just two axes [318, 319, 330]. In the following we consider some simple transfor¬ 
mations, so called quantum gates [331]. 


23.4.6.1 Pauli-gates 

Of special interest are the gates represented by the Pauli matrices U — er, since any 
complex 2x2 matrix can be obtained as a linear combination of the Pauli matrices 
and the unit matrix (Sect. 14.15). For all three of them det U = — 1 and ip = tt. 

The X-gate 


Ux = a x = ' () \ (23.298) 

corresponds to rotation by tt radians around the x —axis (23.291 with 0 (l = 0) 

/ 1 0 0 
Ax = I 0 —1 0 
\0 0 -1 

It is also known as NOT-gate since it exchanges the two basis states. Similarly, the 
Y-gate 


(23.299) 


Uy = a y = (i o') 

rotates the Bloch vector by tt radians around the v—axis (23.291 with <P () = tt/2) 
-1 0 0 \ 

0 1 0) (23.300) 

0 0 - 1 / 
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and the Z-gate 


^=^=(L°i) 

by 7 r radians around the z —axis 


A z = 



(23.301) 


(23.302) 


This rotation can be replaced by two successive rotations in the xy —plane 


Az = A x A y . (23.303) 

The corresponding transformation of the wavefunction produces an overall phase 
shift of V 2 since the product of the Pauli matrices is a x a y = i a z , which is not 
relevant for observable quantities. 


23.4.6.2 Hadamard Gate 


The Hadamard gate is a very important ingredient for quantum computation. It trans¬ 
forms the basis states into coherent superpositions and vice versa. It is described by 
the matrix 



with det U H = — \ and 


/0 0 1 
A h = ( 0 -1 0 
\1 0 0 

which can be obtained as the product 


(23.304) 


(23.305) 


'0 0 -1 
A h = ( 0 1 0 
1 0 0 



(23.306) 
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of a rotation by 7r radians around the x —axis and a second rotation by Y 2 radians 
around the y-axis. The first rotation corresponds to the A"-gate and the second to 
(23.292) with <Pq = Y 2 

t/ = ( Ylfl (23.307) 

V vT -Jl) 


Problems 

Problem 23.1 Wave Packet Motion 

In this computer experiment we solve the Schroedinger equation for a particle in 
the potential V(x) for an initially localized Gaussian wave packet ip(t — 0, x ) ~ 
exp(— a(x — xq) 2 ). The potential is a box, a harmonic parabola or a fourth order 
double well. Initial width and position of the wave packet can be varied. 

• Try to generate the stationary ground state wave function for the harmonic oscil¬ 
lator 

• Observe the dispersion of the wave packet for different conditions and try to 
generate a moving wave packet with little dispersion. 

• Try to observe tunneling in the double well potential 

Problem 23.2 Two-state System 

In this computer experiment a two-state system is simulated. Amplitude and fre¬ 
quency of an external field can be varied as well as the energy gap between the two 
states (see Fig. 23.9). 

• Compare the time evolution at resonance and away from it 

Problem 23.3 Three-state System 

In this computer experiment a three-state system is simulated. 

• Verify that the system behaves like an effective two-state system if the intermediate 
state is higher in energy than initial and final states (see Fig. 23.13). 

Problem 23.4 Ladder Model 

In this computer experiment the ladder model is simulated. The coupling strength 
and the spacing of the final states can be varied. 

• Check the validity of the exponential decay approximation (see Fig. 23.15) 
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Problem 23.5 Semiclassical Approximation 

In this computer experiment we study the crossing between two states along a nuclear 
coordinate. The time dependent Schrodinger equation for a wave packet approach¬ 
ing the crossing region is solved numerically and compared to the semiclassical 
approximation. 

• Study the accuracy of the semiclassical approximation for different values of cou¬ 
pling and initial velocity 

Problem 23.6 Landau-Zener Model 

This computer experiment simulates the Landau Zener model. The coupling strength 
and the nuclear velocity can be varied (see Fig. 23.19). 

• Try to find parameters for an efficient crossing of the states. 

Problem 23.7 Resonance Line 

In this computer experiment a two-state system with damping is simulated. The 
resonance curve is calculated from the steady state occupation probabilities (see 
Figs. 23.23, 23.24). 

• Study the dependence of the line width on the intensity (power broadening). 


1 

- 

I 

1 

1 

/ 

\\ ~~ --___ 

\ \ 

' \ _ 

X 

\ '\ s 

/-----" 


- 

_1_ 1 _1_ 1 _1_1_1_ 


0 10 20 30 40 50 

time 


Fig. 23.27 (Generation of a coherent mixture by a ^-pulse) The equations of motion of the Bloch 
vector (23.268) are solved with the 4th order Runge-Kutta method for an interaction pulse with 
a Gaussian shape. The pulse is adjusted to obtain a coherent mixture. The influence of dephasing 
processes is studied. 7) = 1000, t p = 0.9, Vo = 0.25. The occupation difference pn — P 22 = Z 
(solid curves) and the coherence |pi 2 1 = \\Jx 2 + y 2 ( broken cur\>es) are shown for several values 
of the dephasing time Ti = 5, 10, 100, 1000 
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Fig. 23.28 (Motion of the Bloch vector during n/2 and it pulses) The trace of the Bloch vector is 
shown in the laboratory system. Left 7r / 2 —pulse as in Fig. 23.27 with T 2 = 1000. Right tt —pulse as 
in Fig. 23.26 with T 2 = 1000 


Problem 23.8 Spin Flip 

The damped two-state system is now subject to an external pulsed field (see 
Figs. 23.26, 23.27, 23.28). 

• Try to produce a coherent superposition state (ir/2 pulse) or a spin flip ( 7 r pulse). 

• Investigate the influence of decoherence. 










Chapter 24 

Variational Methods for Quantum Systems 


The variational principle states, that the energy expectation value of any trial function 
is bounded from below by the exact ground state energy. Therefore, the ground state 
can be approximated by minimizing the energy of a trial function which involves 
certain parameters that have to be optimized. In this chapter we study two different 
kinds of quantum systems. First we apply the variational principle to one- and two- 
electron systems and calculate the ground state energy of the Helium atom and the 
Hydrogen molecule. If the trial function treats electron correlation explicitly, the 
calculation of the energy involves unseparable multidimensional integrals which 
can be efficiently evaluated with the variational quantum Monte Carlo method. In 
a second series of computer experiments we study models with a large number of 
variational parameters. We simulate excitons in a molecular aggregate which are 
coupled to internal vibrations. The number of parameters increases with the system 
size up to several hundred and the optimization requires efficient strategies. We use 
several kinds of trial functions to study the transition from a delocalized to a localized 
state. 

The variational principle is a very valuable tool to approximate the groundstate 
energy and wavefunction. Consider the representation of the Hamiltonian in a com¬ 
plete basis of eigenfunctions [277] 



(24.1) 


n 


with the groundstate energy 


(24.2) 


and a trial function with some adjustable parameters 


Ip triad A). 


(24.3) 
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The expectation value of the Hamiltonian 

< 'tptrialHlptrwl >— ^ I < 4>trial\4>n > fe > Ep ^ | < ^trial^n 


= Eq < 4>trial 


^\i>n >< 4’n\ 


4>trial >= E 0 \ljj tr i a i \ 2 . 


(24.4) 


Hence the energy expectation value is bounded from below by the groundstate energy 

< > 


— Eq . 

\4>trial? 

For the exact groundstate 

< iPq\h\4>q > 

IV’ol 2 

and the variance 

2 < 4 J o\H~\ipo > (< V’ol# l^o > 


(24.5) 


= £n 


IV’ol 2 


_ / < 4)q\h\iI>q > \ 

\ IV’ol 2 / 


= 0. 


(24.6) 


Now, let us try to find an approximate solution of the eigenvalue problem 
Hip = Eq4> (24.7) 

by optimizing the trial function. The residual is 

R = Htptriai - £oVw (24.8) 

and, applying Galerkin’s method (p. 272) we minimize the scalar product 


< V trialR > = < t/W/^VW > ~Eq < Ip trial\^ trial > (24.9) 

where the trial function should be normalized. Alternatively, we divide by the squared 
norm and minimize 


< IptrialHllltrial > 
< VwlV "trial > 


(24.10) 


Hence the “best” trial function is found by minimizing the energy with respect to 
the parameters A. 
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Now, assume that the groundstate is normalized 

I*/> 0 | 2 =1 (24.11) 


and choose the normalization of the trial function such that 


Atrial = 4>0 + P 

< ip 0 \p >= 0 . 


Then, 

< Ip trial trial > E 0 + < pH p > 2 

= 1 + \p\ 2 =Ea + ° W) 

the accuracy of the energy is of second order in \p\. From 

< Ip trial Ip trial > _ £p+ < pH 2 p > 

I Iptrial 1 2 1 + \P\ 2 

we find that the variance of the energy 


(7 


2 

E 


< Iptrial Ip,rjal > _ / < Ip trial H Ip tria l > \ 2 

\lp trial] 2 \ I Ip trial 1 2 / 


_ El+ < pH 1 p > / E 0 + < pHp >\ 2 

i + |p | 2 V 1 + lPl 2 / 

~ £0(1 - lp| 2 )+ < pH 2 p > -£o(! - 2|p| 2 ) - 2Eo < pHp > 
^ Eq < p|p > 2 + < pH\Hp > — 2£o < pElp > 

* |(//-£o)p| 2 


(24.12) 

(24.13) 


(24.14) 


(24.15) 


(24.16) 


is also second order in \p\. It is bounded from below by zero. Therefore, Quantum 
Monte Carlo methods often minimize the variance instead of the energy for which 
the lower bound is unknown. 


24.1 Variational Quantum Monte Carlo Simulation 
of Atomic and Molecular Systems 

Electron structure calculations for atoms and molecules beyond the self consistent 
field level (Hartree Fock uses one Slater determinant as a trial function, MCSCF 
methods a combination of several) need an explicit treatment of electron correla¬ 
tion. This can be achieved by expanding the wavefunction into a large number of 
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configurations (Cl method) or, alternatively, by using trial functions which depend 
explicitly on the electron-electron distances. Very popular [332, 333] are factors of 
the Jastrow pair-correlation [334] type 


exp 


Z^> 


(24.17) 


where in the simplest case 


U( rij ) = 


ar n 


1 + Pry 


(24.18) 


has the form of a Pade approximant. Wavefunctions including a Jastrow factor do 
not factorize and make it necessary to apply Monte Carlo integration methods to 
calculate the energy expectation value (see p. 205). For the computer simulation of 
two-electron systems we use trial functions of the type 


^ _ Q—nria Q—Kr2b gQiri2/(l+/3ri2) 


(24.19) 


which are products of two ls-orbitals centered at the (possibly same) positions r ab 
and a Jastrow factor. In the following, we abbreviate 

u = 1 + pr 12 . (24.20) 


Starting with the derivatives 


d , 
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ip + 
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>'\ 2 u- 


ax¬ 
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a(3x 


3 „2 


r h «• 


r 2 n u 3 j 


ip 


(24.21) 


(24.22) 


we calculate the kinetic energy 


Tip=--(V 2 + V 2 )ip- 


2a 


aK, 

( *\a 

rib' 

T P1 


\r\a 

r 2b > 

r l2. 




+ 


K K 

ria r 2b 


rnu- 


M 3 


Ip. 


(24.23) 


For short electron-electron distance 
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(24.24) 


A choice of a = 1/2 cancels the divergent Coulomb repulsion at r \2 -> 0 and 
fulfills the electron-electron cusp condition [333, 335]. More complicated Jastrow 
factors also allow to fulfill the electron-nuclei cusp conditions. 


24.1.1 The Simplest Molecule: 

As a first example (Problem 24.1), we consider an electron moving in the Coulomb 
field of two protons (Fig. 24. 1). Applying the Born-Oppenheimer approximation the 
protons are kept fixed at a distance R. In atomic units, 1 the Hamiltonian is 

1,111 

H=T+V = —V 2 -P-. (24.25) 

2 r a r b R 

This eigenvalue problem can be solved exactly (using elliptic coordinates) and is 
also a popular example for the variational method. 

As a trial wavefunction we use the linear combination of two hydrogen-like 1 s 
orbitals 


Wa = 




(24.26) 


which are solutions for the problem with two nuclear charges n at infinite distance. 
At finite distances, the variational parameter k is a measure of the effective nuclear 
charge. For large distance k = 1 as for a single proton whereas at short distances the 
optimum value approaches k — 2 as for the He + ion. 

Since the problem is highly symmetric,we take a symmetric combination 


1 i.e. setting ag = 4neoh 2 /e 2 m e = l and h 2 /m e = 1. 
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1 


trial = 


VPa ± (fi b \ 


V2(l ± S) 

where the overlap integral can be calculated using elliptic coordinates 

R R 

r a = ~(X + R) r b = - (A - n) 


(24.27) 


S = 


/OO PI 

d\J dji 2k 3 (A 2 — R 2 )— e_Ki?A 


2d2 


= e _KS (l + kR+ ^^). 


k z R 


The action of the Hamiltonian is 


(24.28) 



r 1 1 1 1 
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<Pa 

(24.29) 

Wb 

(24.30) 


V2(l ±5) 

from which we obtain the local energy 
Eloc — 


K K 2 

f K K 2 \ 
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1 ] 

(---)^ a ± 

L ^ 2 
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_R 

r a 

r b_ 


Atrial 


' 1 
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1 

At 2 ] 

! T.Ve^tyb 


r a 


2 _ 

Pa ± Pb 


1 ±s 


For comparison, we calculate the expectation value of the energy 

1 H aa ± //„/, 

< ‘fitrialHtptrjal >= ± \H aa + H bb ± H ab ± H ba \ = 

with the matrix elements 

1 K 2 f ip 1 f ip 2 

H aa = H bh =--—- / ^ dV + (K - 1) / ^dV 
R 2 J r h J r a 




^iV + («-l)/^. 

r a J r b 


(24.31) 

(24.32) 

(24.33) 

(24.34) 
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The integrals can be evaluated in elliptic coordinates 


<Pb 


vi 


R z 


—dV = / —dV = 2^ / dX / dfj, — ( A-/n)e 


— kR(\+h)/2 _ 


n 






R 1 


L±dV = —dV = d X dfj, — (X + 


— k,R(\-\-/i )/2 


n 


=- s~ 2 kR (k+ -) 

R R 


f ™^-dV = [ V^-dV = 2k 3 [ dX [ (A-/i)e" KfiA 
J r b J r a J i 7-t 4 


= e 


(k + . 


(24.35) 


(24.36) 


(24.37) 


In our computer experiment (Problem 24.1), we first keep k — 1 fixed and use the 
variational MC method to calculate the expectation value of the energy. Figure 24.2 
compares the results with the exact value (24.32). Next we vary k and determine 
the optimum value at each point R by minimizing R(R, k). Figure 24.3 shows the 
K-dependence for several points. The optimized /,--values (Fig. 24.4) lead to lower 
energies, especially at short distances. The equilibrium is now atRo = 2.0 Bohr with 
a minimum energy of —0.587 a.u. instead of 2.5 Bohr and —0.565 a.u. for k = 1. 
(Exact values are 2.00 Bohr and —0.603 a.u. [277]). 



the variational method. Circles show the results from MC integration for a maximum step length 
of 0.5 Bohr and averages over 2 x 10 7 samples for a fixed effective charge k = 1. The solid curve 
shows the results of the exact integration (24.32) for comparison. Diamonds show the MC results 
after optimizing n(R) at each point. The dashed curi’e shows the results of the exact integration 
(24.32) where k(R) was determined by solving trial Retrial >= 0 numerically 
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Fig. 24.3 (Optimization of 
the variational parameter k 
for Ht) The groundstate 
energy from MC integration 
is shown as a function of k 
for R = 1 (diamonds), R = 2 
(squares) and R = 3 
(circles). The curves show a 
fit with a cubic polynomial 
which helps to find the 
minima 


Fig. 24.4 (Optimized 
effective charge parameter 
for Ht) The variational 
parameter k is optimized by 
minimizing the MC energy 
as shown in Fig. 24.3 
(circles). The curve shows 
the exact values obtained by 
minimizing (24.32) 
numerically 


Fig. 24.5 Geometry of He 





24.1.2 The Simplest Two-Electron System: The Helium Atom 


The Helium atom (Fig. 24.5) is the simplest “many-electron” system where electron- 
electron interaction has to be taken into account. The electronic Hamiltonian reads 
in atomic units 



2 1 

— H-■ 

r 2 02 


(24.38) 
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Without electron-electron interaction, the singlet groundstate would be simply 
given in terms of hydrogen-like ls-orbitals as 


2 3 


V’o = — e 'e 2 —= (f (1) ^ (2)- f (2) | (1)) 
7T V2 


(24.39) 


with 


I 2 1 2 2 
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1-) -00 — 2 ( 
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r 2. 

n/ V 
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2 2 

— i’O —1 
r l r 2 


(24.40) 


For the variational treatment (Problem 24.2) we use a trial wavefunction with a 
variable exponent to take the partial shielding of the central charge into account 

VW = -e—'e—=(f (1) | (2)- f (2) 4 (1)) (24.41) 

7T V2 

where the antisymmetric spin function accounts for the Pauli principle. 

Then, 

1 / , 2k , 2k\ /I 2 2 \ 

Hlptrial = ~ - ( K 2 -hC-) Vf rial + (-I Atrial (24.42) 

2 \ n r 2 / \ri 2 n r 2 J 


1 , k — 2 k — 2 

Eloc =- k H-1-. 

r 12 r i r 2 

The integration can be performed analytically [277]. First we calculate 

3. 2 3 

r e -2wi e _2 Kr2 -dV x dV 2 = — [ e~ 2Kr -dV = k. 


(24.43) 
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(24.44) 


The integral of the electron-electron interaction is 
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f r\dne- 2 ^ (47t) 2 i p r\dr 2 Q 

poo 

16 k 6 / rrdrie~ 2Kn 

] Q—2nr\ g— 2kt\ ~ 

4r\K? 4tt 2 

Jo 1 
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— K. 



r 2 dr 2 e 


—2kt2 


(24.45) 


Together, we obtain 


< Atrial Hi/},rial >= ~ K~ + - K + 2(« - T)K = K~ 



which has its minimum at (Figs. 24.6 and 24.7) 


l^min 



1.688 


with the value 


(24.46) 


(24.47) 


min* < ^trialH ^ trial >= ~ ^ “2.848. 

256 

Next, we consider a (not normalized) trial wavefunction of the Slater-Jastrow type 
(24.19) 


Vw = e -Kri e~ Kr2 e“'' 12/(1+/3 '' 12) 


1 

7 ! 


(t (l) 4- (2)- t (2) 4 (D) • 


(24.48) 


Fig. 24.6 (Optimization of 
the effective charge for the 
Helium atom) The 
groundstate energy of the 
Helium atom was calculated 
with MC integration. The 
circles show the average over 
10 7 points. The curve shows 
the exact result (24.46) for 
comparison. The optimum 
value is K = 1.688 
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Fig. 24.7 (Standard deviation of the MC energy) The Circles show the standard deviation of the 
MC energy for Helium. Its minimum between k = 1.7 ■ ■ ■ 1.9 is close to the minimum of the energy 
(Fig. 24.6). The curve shows a cubic polynomial fit 
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Fig. 24.8 (Variation of (5) The groundstate of the Helium atom is approximated with the Slater- 
Jastrow wavefunction (24.48). Singularities of the potential energy are removed by using k = 2 and 
a = 1/2. Each point represents an average over 10 7 samples. Left The energy minimum of—2.879 
is found at /? = 0.15. Right the standard deviation has a minimum value of 0.29 at )3 = 0.35 


From (24.23) with r a = r/, we find the local energy 


(24.49) 

With fixed values a = 1 /2 and k = 2 all singularities in the local energy are 
removed, but this also reduces the flexibility of the test function. The energy minimum 
of—2.879 is found at (3 = 0.15 (Fig. 24.8). A further improvement can be achieved by 
varying the exponent k together with (3. The minimum now is —2.885 at k = 1.91 
(Fig. 24.9). If we drop the cusp condition and vary all three parameters we find a 
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Fig. 24.9 (Variation of k) The groundstate of the Helium atom is approximated with the Slater- 
Jastrow wavefunction (24.48). From Fig. 24.8 the optimized value of (3 = 0.15 is taken, a — 1/2. 
Each point represents an average over 10 7 samples. Left The energy minimum of —2.885 is found 
at k = 1.91. Right the standard deviation has a minimum value of 0.33 at k = 1.98 



Fig. 24.10 (Variation of all parameters) The groundstate of the Helium atom is approximated with 
the Slater-Jastrow wavefunction (24.48). Variation of all three parameters gives a lowest energy of 
—2.891 for a = 0.38, /J = 0.18, k = 1.85 


slightly smaller value of —2.891 with a standard variation of a = 0.36 (Fig. 24.10). 
More sophisticated trial wavefunctions reproduce the exact value of —2.903724 even 
more accurately [336, 337], 


24.1.3 The Hydrogen Molecule H 2 

The Helium atom can be considered as the limiting case of the H 2 molecule for 
zero distance (neglecting nuclear Coulomb repulsion). At finite distance R the one- 
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electron factors of the wavefunction have to be symmetrized. 2 We use a trial function 
(we omit the singlet spin function and do not normalize the wavefunction) 

V’ = C lVa( r i)ipb(r 2 ) + (Pb(ri)ip a (r 2 )] + (1 - C) [(p a (n)<p a (r 2 ) + tpb(n)Vb(r 2 )] 
= C^ VB + (1 - C)Vw (24.50) 

which combines covalent and ionic configurations 

4>vb = (Wa(r\)<Pb(r2) + <Pb(n)Va(r2)) (24.51) 

bionic = {Wa(r\)Wa(r2) + <MQ)<Mr 2 )) (24.52) 

and includes as special cases 

• the Heitler-London or valence-bond ansatz (C = 1) ipvB 

• the Hund-Mulliken-Bloch or molecular orbital method where the symmetric MO 
is doubly occupied (C = 0.5) 

^mo = (Vairi) + ifb(ri)) (<p a (r 2 ) + tp b (r 2 )) 

= fl’VB + bionic (24.53) 

• the Heitler-London method augmented by ionic contributions C = (1 + A) -1 

^ = IpVB + tyionic (24.54) 


• the MCSCF ansatz which mixes two determinants (C = 1 — C,i) 

^ ~ ^MO + C d’ t l J MO 

= (^(n) + <Pb(n)) (vain) + <Pb(n)) + Q (v> a {n) - <Pb(n)) (vain) - <Pb( r 2)) 
= (1 - C d W VB + (1 + C d )4> ionic . (24.55) 

The molecular orbital method corresponds to the Hartree-Fock method which is 
very popular in molecular physics. At large distance it fails to describe two separate 
hydrogen atoms with an energy of — 1 au properly. In the bonding region it is close 
to the valence bond method which has the proper asymptotic limit. Both predict an 
equilibrium around R = 1.6 (Fig. 24.1 1). 

To improve the results we vary the effective charge k and the configuration mixing 
C (Fig. 24.12). Optimization of k lowers the energy especially at small internuclear 
distances where the effective charge reaches a value of 2 as for the Helium atom 
(Fig. 24.13). The minimum of the potential curve now is found at a much more 
reasonable value of R — 1.4. Variation of the configuration mixing lowers the energy 


2 We consider only singlet states with antisymmetric spin part. 
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Fig. 24.11 (Comparison of Heitler-London and Hund-Mulliken-Bloch energies for H 2 ) The MO 
method ( circles ) fails to describe the asymptotic behaviour at large distances properly. In the bonding 
region it is close to the VB method (squares). Both predict a minimum around R = 1.6 



Fig. 24.12 (Optimization of effective charge k and configuration mixing C ) Starting from the MO 
energy (black circles) optimization of k (red squares) and C (blue diamonds) lower the energy 
considerably and shift the potential minimum from 1.6 to 1.4 Bohr (see Problem24.3) 


mostly at larger distances where the proper limit is now obtained. For our computer 
experiment (Problem 24.3) we include a Jastrow factor into the trial function 


4> = {C [ e - Kr '-~ K ^ 


j 0 b ^2 a 


] + (1 - C) [ e ~ K '' lo-K '' 2 ° 


_j_ Q-Krib-K,r 2 b 


]} 


x exp 


urn 

1 + fir 12 


(24.56) 


and vary n, )3 and C to minimize the expectation value of the local energy 
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Fig. 24.13 (Optimized values of k and C) The effective charge ( squares ) approaches k = 2 at very 
short distances corresponding to the He atom. Configuration mixing ( circles ) is most important 
in the bonding region. At large distances the valence bond wavefunction (C = 1) provides the 
lowest energy. At very small distances the two configurations become equivalent making the mixing 
meaningless as R —> 0 



1 

< 

2 a\ 

1 

i 

Eloc — 

— 1 

1 

— 




I'll 

( 

U L ) 

ria 

rib 

+ c 

Hi 

Hi 

KOL 

( ri a 

rib 

—i 


H- T 

— 



L ria 

rib 

u 2 

\ria 

rib 

+ c 

Hi 

Hi 

Ka 

( rib 

ria 

—1 


H —w 

— 



L n b 

ria 

u 2 

\ r\ b 

ria 

+ (1 

- c ) 

Hi 

K 

Hia 

/ r ia 

— 

■a - 

H- 2 



L n a 

ria 

u 2 

\ria 

+ (1 

~C) 

Hi 

Hi 

na 

( rii 

— 

H- 

+ ~~2 

— 



L r\b 

rib 

U 2 

V rib 


1 1 

ria rib 
(Ti ~ r 2 )~ 

n i . 

(Ti - r 2 )~ 
r 12 


2a(3 

3~ 


or 

u 


Qarn/u-Kria-K^bjjj-l 

e ari 2 /u-K,ri b -Kr 2a Jj-l 


r2a\ (IT - r 2 ) 

ria ) r 12 

iT*\ (r t - r 2 ) 

rib ) r 12 


garn/u-Kru-nriajj-l 

e ar n /u-K,r lb -K.r 2 i, x p-l 

(24.57) 


In the bonding region the energy is lowered by further 0.01 au with a minimum value 
of — 1.16au (Fig. 24.14). This effect is small as part of the correlation is already 
included in the two-determinant ansatz. More sophisticated trial functions or larger 
Cl expansions give -1.174 a.u. quite close to the exact value [333], 


24.2 Exciton-Phonon Coupling in Molecular Aggregates 

In this section we simulate excitons in a molecular aggregate which are coupled to 
internal vibrations of the molecular units. Molecular aggregates are of considerable 
interest for energy transfer in artificial [338] and biological systems [339]. Even 
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Fig. 24.14 (Optimization of 
the Slater-Jastrow 
wavefunction for Hj) 
Optimization of the Jastrow 
factor lowers the energy by 
further 0.01 au ( triangles ). 
Circles show the MO energy, 
squares the MO energy with 
optimized exponent k and 
diamonds the optimized 
MCSCF energies as in 
Fig. 24.13 



distance R (Bohr) 


simple trial functions involve a large number of parameters which have to be opti¬ 
mized and require efficient strategies to minimize the energy. We consider a finite 
periodic system like in the light harvesting complex of photosynthesis. An optical 
excitation on the n-th molecule is denoted by the state |n >. It can be transferred to 
the neighboring molecules by the excitonic coupling V and is coupled to the vibra¬ 
tional coordinate q n . (For simplicity, we consider only one internal vibration per 
molecule). The model Hamiltonian reads in dimensionless units (periodic b.c. imply 
that |0 >= |N > and \N + 1 >= |1 >) 


n = 'y' \m > H mn < n\ 


= — + 




> A q n <n\ + \n>V < n + \ \ + \n > V < n — 1|. 


(24.58) 


Due to the N-fold degeneracy of the excited states, a simple Born-Oppenheimer 
wavefunction is not adequate. Instead we consider a sum of N Born-Oppenheimer 
products 

V = y. \n > - q N ). (24.59) 

n 

We use the variational principle to approximate the lowest eigenstate. Obviously, 
the number of variational parameters will rapidly increase with the system size. Even 
if we introduce only one parameter for each unit, e.q. a shift of the potential minimum, 
this requires N 1 parameters for the aggregate. 

The Hamiltonian (24.58) can be brought to a more convenient form by a unitary 
transformation with 
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s = ^ \n > G n < n\ (24.60) 

n 

where the translation operator G transforms the nuclear coordinates according to 
Gq n G~ l — q n+ \. (24.61) 

The transformed Hamiltonian then reads 


1 d 2 1 A 

T[-2^ + - 2 r») 


= — + \ + ot/n ) + Ago + |n> VG <n+l| + |n> VG 1 < n - 1|. 


Delocalized exciton states 
14 


(24.62) 


(24.63) 


transform the Hamiltonian into N independent exciton modes 

H = ^\k > H k < k\ (24.64) 

k 

with 

H k = y + X + ^0 + Ve ik G + Ve^G" 1 . (24.65) 

Hence, we conclude that the eigenfunctions of the Hamiltonian H have the general 
form 


* = -U y e iA '"|« > G"0 k 


where <0 k is an eigenfunction of H k and the number of parameters has been reduced 
by a factor of N (since for each k, only one function <P k is involved). 

In the following we study the lowest exciton state, which for V < 0 is the lowest 
eigenfunction 3 for k = 0. Hence, the wavefunction of interest has the form 


^ = -U V I n> G"<P 

vwr 


(24.66) 


and can be chosen real valued. 


3 This is the case of the so called J-aggregates [338] for which the lowest exciton state is strongly 
allowed. 
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24.2.1 Molecular Dimer 


To begin with, let us consider a dimer (N = 2) consisting of two identical molecules 
in a symmetric arrangement. The model Hamiltonian reads in matrix form 


1 <9 2 1 d 2 


2 dq\ 2 dq\ 


o(qi + A)" + ~q~ 


V 


V \q\ + \{qi + A) 2 _ 

and can be considerably simplified by introducing delocalized vibrations 

< 7 i ± qi 


<7± 


V2 


(24.67) 


(24.68) 


which separates the symmetric mode q + 


Y-i — 

V 2dc i\ 


H - I 77T - + o(9+ + I “ + 


) 2 dqz_ 


i (‘I- + 7s) J v 


The lowest eigenfunction of the symmetric oscillation is 

1 A 


0 O+ = 7 r exp 


(<Z+ H-7=) 


' V2 

with the eigenvalue (the zero point energy) 

1 

E 0+ = -. 


(24.69) 


(24.70) 


(24.71) 


Hence, for the dimer we may consider a simplified Hamiltonian with only one 
vibration 


1 1 d 2 

H= -- 

2 2dq 2 


( 9+ 

V 


it 


V 

q ~ 72 ) J 


(24.72) 


According to (24.66) the k = 0 eigenstates have the form 


0 = -J=<Z>|1 > + -J=G<Z>|2 >= ~^j=0(q)\\ > +-^=0(-q)\2>. 


(24.73) 


For the dimer problem, the eigenstates can be calculated numerically by diago- 
nalization of the Hamiltonian equation (24.72) in the basis of harmonic oscillator 
states [340]. 
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Dressed Exciton 

The simplest trial function which is well known as “dressed exciton” or “mean field” 
ansatz [341, 342], uses a Gaussian representing the groundstate of a displaced (and 
possibly distorted) harmonic oscillator 

0 = 1 — J e ~K(q-+a) q 0 = f_\ e -«(«_- a ) (24.74) 

with the energy expectation value 

E mf (k, A) =< >= I + (-L + ^) + h a - 4=) 2 + Ve" w (24.75) 

2 8k 2 2 y/2 

for which the first and second derivatives are easily found 


8Emf A -'W 

—-— — a -— — 4Fo;Ke 

da V 2 

(24.76) 


(24.77) 

d2 ^ MF = i + 4VKe- W [Ana 2 ll 
dor 

(24.78) 


(24.79) 

= 4Fae- W [2k« 2 l] . 

OKOa 

(24.80) 

In the limit of vanishing “dressed” coupling Ve. ~ ' 

so called self trapped state, the lowest energy is found for 

« 0, corresponding to the 
a = A/a/2, k = 1/2 

min Emf ( V e _2K “ 2 0) = 1 

(24.81) 

which is the zero point energy of the two dimer modes. In the limit of vanishing 
exciton-phonon coupling A = 0 (the fully delocalized state) the energy is minimized 
for a — 0, k = 1/2 

min£AfF(A —*■ 0) = V + 1. 

(24.82) 


For the general case we apply the Newton-Raphson method (p. 124) to locate the 
minimum. It is quite important to use a reasonable starting point to ensure conver- 
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Fig. 24.15 (Variational solutions for the dimer) The lowest excitation energy of the dimer Hamil¬ 
tonian is shown as a function of the reorganization energy A 2 /2. The mean field ansatz (red curi’es) 
predicts a sharp transition to the self-trapped state and deviates largely for A 2 /2 > 5. Variation of 
the exponent k improves the agreement in the transition region considerably (full red curve ) as 
compared to the standard treatment with fixed k = 1/2 (dashed red curve). The black curve shows 
the numerically exact solution for comparison 


gence to the lowest energy. 4 In Problem 24.4, we search for an approximate min¬ 
imum on a coarse grid first. Figure 24.15 shows the calculated energy minimum 
for strong excitonic coupling V = — 5 as a function of A 2 . For small values of the 
exciton-phonon coupling, the numerically exact values are reproduced quite closely. 
For larger values the mean field ansatz predicts a rapid transition to a so called 
self-trapped state [343] with a — \/s/2 and a very small Franck-Condon factor 
F — exp(—2 na 2 ) ~ 0 (Figs. 24.16, 24.17). In this region, the deviation from the 
numerical exact result is appreciable, especially if only a is varied and n = 1 /2 kept 
fixed. 

Solitonic Solution 

In the region of large exciton-phonon coupling a simple ansatz similar to Davydov’s 
soliton [344] is quite successful (Fig. 24.1 8) which breaks the symmetry of the system 
and uses a trial function 


'I'soi = (V=»i 11 > +¥>2|2 >)0 aua2 (q i, qi) (24.83) 

with two mixing amplitudes with the constraint 

y\ + ip\=l (24.84) 


4 In the transition region, the energy may converge to an unstable state, depending on the starting 
point. 
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Fig. 24.16 (Optimized 
parameters of the mean field 
ansatz) The optimized 
parameters for V = — 5 show 
a sharp transition to the self 
trapped state. Full curves 
optimization of a and k. 
Dashed cun’es optimization 
of a for fixed k = 1/2 


Fig. 24.17 (Franck-Condon 
factor of the mean field 
ansatz) The transition to the 
self trapped state shows also 
up in the Franck-Condon 
factor F = exp {—2 ko: 2 } 
which is shown in a 
semi-logarithmic plot. Full 
curve optimization of a and 
k. Dashed curve 
optimization of a for fixed 
k = 1/2. The dotted curve 
shows the numerically exact 
result for comparison 




Fig. 24.18 (Variational 
solutions for the dimer) The 
soliton approach (dashed 
blue curve) works quite well 
for large but also for very 
weak exciton-phonon 
coupling. The delocalized 
soliton interpolates between 
mean field and soliton results 
and describes the transition 
quite well (red curve). The 
black curve shows the 
numerically exact solution 
for comparison 
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and the same vibrational function for both states (in the self trapped region distortion 
of the oscillator is not important) 


1 

qi) = 


-Qi+qi) 2 /2 -(^ 2 +«2) 2 /2 


(24.85) 


The energy expectation value is 


EsoiifPu <P2, at, a 2 ) =< VH'l' > 

(ai — A ) 2 a}, 

_ _ — 

2 2 


= <p! 


1 + 


1 + 


(a 2 - A ) 2 a\ 


■ ZVipiipi 


— 1 + ^(ct i — <£ 2 A) + -(a 2 — <^;A) 2 + A 2 ^ 2 ^ + 2 V(/ji</j 2 (24.86) 


and for the optimized values 

< = ^ 

it becomes 


(24.87) 


Esoltown, ot°,a%) = 1 + A 2 QfQ 2 + 2 ' / V 2 l¥ 2 2 = 1 + -4- |2qi^2 + 


2V\ 2 V 2 
A 2 "/ ~~ A 2 "' 
(24.88) 


Alternatively, using symmetrized coordinates we obtain 


E so i(ipi,ip2,a + ,a-) = 1 + ^ ^a+-+ \\<X- ~ (Vi ~ ‘fib ) + + 2Vtpiip 2 


(24.89) 


and optimum values 

a°_ = ~ $)• 


(24.90) 

(24.91) 


Since |2y>iy> 2 | < 1, the minimum for large exciton-phonon coupling is at the bottom 
of the parabola 


V 2 , 

min E sot = 1 - — for | V| < \~/2 


(24.92) 
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whereas in the opposite case it is found for 2ip\ip 2 = 1 (V is assumed to be negative) 


A 2 2 

min E so i = l + V + — for | V | > A 2 /2. 

(24.93) 


The transition between the two regions is continuous with 


min E so i = 1 + y for | V| = A 2 /2. 

(24.94) 

Delocalized Soliton Ansatz 



Mean field and soliton ansatz can be combined by delocalizing the solitonic wave 
function [345]. The energies of the trial function 


Vsol = (v?l 11 > +y? 2|2 >)0 

(24.95) 


and its mirror image 


^ ol =(<p 2 \l >+Vi\2>)G& 

(24.96) 


are degenerate. Hence delocalization of the trial function 


'Fdelsol = |1 > + WlG®] + 2 > [lf2 < l > + <PlG<P] 

(24.97) 


is expected to lower the energy further and ensures the proper form of (24.73). Its 
norm is 


< ^delsoll^delsol >— 2(1 + 2(fiiip 2 F) 

(24.98) 


with the Franck-Condon factor 


F =< <P\G\<P >= e~ K Gi-<*2) 2 = e -W_. 

(24.99) 

The expectation value of the Hamiltonian simplifies due to symmetry 



< ^delsol^^delsol > = 2 </?i < > +2<p\ < <PH 2 <P > +4(pi<p 2 < <PH[G<P > 


+2V [F + 2ipnp 2 \ ■ 

(24.100) 

Finally, varying only the antisymmetric mode, the energy is 



l ^2 % - (ip\ - tfya-X + 2ip\ip2 r-2F/t 2 a?_ + vl + VF 

_I_I_(=_J_ 


K 1 A 

Edelsol — X + o -f 77" + 

2 8k 2 


1 + 2<pi ip 2 F 


(24.101) 
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In Problem 24.5 we locate the minimum energy with the steepest descent (6.2.5) 
or the conjugate gradient (6.2.5) method 


24.2.2 Larger Aggregates 

The variational methods for the dimer can be generalized for larger systems [345]. 
The mean field ansatz for the lowest k = 0 state becomes 



(24.102) 


with 


N 


0 = J ^ 7 r - 1 / 4 e - fe +«») 2 /2 


(24.103) 


The energy is 



n 


=< 0H O 0 > +2VF 



(24.104) 


n 


and its gradient 


rx — “1“ VF(2a n OL n - |_i OL n — l)* 

UOt n 


In Problem 24.5 we locate the minimum energy with the steepest descent (6.2.5) 
or the conjugate gradient (6.2.5) method. As for the dimer, the mean field method 
shows a rapid transition to the self-trapped state. The starting point is quite important 
as in the vicinity of the transition the gradient based methods eventually converge to 
a metastable state (Fig. 24. 19). 

The soliton wavefunction (corresponding to Davydov’s D1 soliton) for the aggre¬ 
gate is 
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Fig. 24.19 (Variational solutions for a 10-mer) The lowest energy of a periodic 10-mer is calcu¬ 
lated for V = —5 (see Problem 24.5). The mean field wavefunction gives (green and blue curves ) 
reasonable values for small values of A 2 and predicts a rapid transition to the self trapped state. 
Approaching the transition from below or above the calculation may end up in a metastable state 
{dashed green and blue curves ). The solitonic wavefunction {dashed black curve ) provides lower 
energies at larger values of A 2 and a much smoother transition to the self trapped state. The delo¬ 
calized soli ton {red curve) gives the lowest energy at all values of A 2 . The zero point energy has 
been subtracted 


with the constraint 



(24.106) 


n 


where <P is given by (24.103). The energy is 


Esol —^ Vsol\H\V sol >— ^ ' ^P n < ^ \H n \0 > T - V ^ ' J (SPn^pn-\- 1 4“ (Pn^Pn— l) 


n 


n 



” + V ^^nVn+l + VnVn-l) 


n 


n n 


N ( a n — A ip 


2 2 


n 



,(</Wn+1 + <PnVn-\) 


(24.107) 


with the optimum displacements 


< = 


(24.108) 
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The energy functional becomes 

NX 2 / \ 

EsoliVn, a° n ) = — + — I 1 - X + 2 V X (PnVn+l (24.109) 

and its gradient 

~w~ = ~2A 2 </^ + 2 V (24.110) 

(Jtpn 

In Problem 24.5 we locate the minimum energy by varying the tp n under the constraint 
(24.106). At larger exciton-phonon coupling, the energy of the soliton wavefunction 
is much lower in energy than the mean held result and the transition to the self- 
trapped state is smoother. At small exciton-phonon coupling, the mean held ansatz 
is lower in energy (Fig. 24.19). 

Similar to the dimer case, the solitonic wavefunction can be delocalized by com¬ 
bining the N degenerate mirror images 

^ip„\n + m > G m & m= l- ■ -N (24.111) 


into the trial function 

'Edelsol = X e lA '" X Vn\n + m > G m & 


m n 


= — ri > G n '-"<P = — £e ifa y > G n '^e- ifa V„G-"<Z>. 

nn' vN n' 

(24.112) 

From the squared norm 

^ *Edelsol\*Edelsol ^ , '~f'n ! "Pd < 0G 0 >= ^ 0 n 0n'En—n' (24.113) 


and the expectation value 

< 'I'delsolH'Edelsol > = jj X < ^G - '" <n + m\ip„Htp n f\m' + n' > G m 0 > 
m,n,m',n' 

= i X < ^G- m ip n H„ +m ip n ,G m+n - n '0 > 

m,n,m',n' 

+ ^7 X V < 0G~ m ip n ‘Pn'G m+n ~ n ' +1 0 > 
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+ ^ Y. V <<PG- m tp n ip n ,G m+n - n '- l <l> > 

m, n,n r 

= ^ < 0G n H o G~ n '0 > 

n,n r 

+ V^<p n p n ' < (PG n ~ n+l 4> > +F^< p n ip n f < <2>G' !_,! ' _1 <2> > (24.114) 

n, n' n,n' 

we obtain the energy of the k = 0 state 
N A 2 

Edelsol — ~ T* 

+(X {pn ^ ni l 

V nn' 

+ V ^.pnPn'O 
nn' 

with the Franck-Condon factors 

F k = < <PG k <P >= e _ 2:'” ( “" _ “" +t)2/4 = -“».“».+«:)/2_ (24.116) 

The results for longer aggregates are qualitatively similar to the dimer. The delo¬ 
calized soliton interpolates between mean field and soliton wave functions and shows 
a smooth transition (Fig. 24.19). 


A (O-n' Olii'—n) -}~ ^ OL m OL m + n i— n k n n 

m J 

-1 ll Pnpn 1 kn—n 1 I 


-«+l + F n '—n—\ 


(24.115) 


Problems 


In the first three computer experiments, we use the variational quantum Monte Carlo 
method to calculate the groundstate energy. The Metropolis algorithm with N w walk¬ 
ers is used to evaluate the integral 


£( k , R) 


< tp K H-ilj K > f 3 

<AA> =! dr 


l*,(r)l 5 


/ \i/j K (r')\ 2 d 3 i 


-Ei oc { r). 


Adjust the maximum trial step to obtain an acceptance ration of about 1 and study 
the influence of the number of walkers on the statistical error. 
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Problem 24.1 

Optimize the effective nuclear charge k for the hydrogen molecular ion lit as a 
function of R and determine the equilibrium bond length. The trial function has the 
form 


Atrial 




Problem 24.2 

For the Helium atom we use a trial wavefunction of the Slater-Jastrow type 


VW = e - Kri e“ Kr2 e“' 12/(1+/5ri2) 


1 


(t (1) i (2)- t (2) 4 (D) 


to find the optimum parameters a , /?, k. 

Problem 24.3 


In this computer experiment we study the hydrogen molecule Hi - The trial function 
has the form 


Vw = {C[e~ Kri -- Kr2b + + (1 - C) + e - Krii ’“ Kr2i ']} 


x exp 


ar 12 


1 + 0r n . 

Optimize the parameters n, 3, C as a function of R and determine the equilibrium 
bond length. 

Problem 24.4 


In this computer experiment we simulate excitons in a molecular dimer coupled to 
molecular vibrations. The energy of the lowest exciton state is calculated with the 
dressed exciton trial function including a frequency change of the vibration 


Atrial 


I 


1 > 





The parameters k, a are optimized with the Newton-Raphson method. Vary the exci¬ 
ton coupling V and the reorganization energy A 2 /2 and compare with the numerically 
exact values. 

Problem 24.5 

In this computer experiment we simulate excitons in a molecular aggregate coupled 
to molecular vibrations. The energy of the lowest exciton state is calculated with 
different kinds of trial functions 





Problems 
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the dressed exciton 


Vmf = X I " > G " II 7 r - 1 / 4 e ~ ( «”+“"> 2 / 2 


n=l 


the soliton 


Viol = 'y, V>n\n > 1~f 7 r 1/4 e (9 " + “" 


) 2 /2 


• the delocalized soliton 

- 7 = YY J Vn\n + m> G m ]^[ 7 r 1/4 e (9 " +Q " ,2/2 . 

-v '-^ m /i n=l 

The system size can be varied from a dimer (N=2) up to chains of 100 molecules. 
The N equilibrium shifts a„ and the N excitonic amplitudes (p„ are optimized with 
the methods of steepest descent or conjugate gradients. The optimized parameters 
are shown graphically. Vary the exciton coupling V and the reorganization energy 
A 2 /2 and study the transition from a delocalized to a localized state. Compare the 
different trial functions. 


Appendix A: Performing the Computer 
Experiments 


The computer experiments are realized as Java programs which can be run on any 
platform if a Java runtime environment (JRE) is installed. They are written in a 
C-like fashion which improves the readability for readers who are not so familiar 
with object oriented programming. The source code can be studied most conveniently 
with the netbeans environment which is open source and allows quick generation of 
graphical user interfaces. The screenshot in Fig. A.l shows an example. 

After downloading and unzipping the zipped file from extras.springer.com you 
have two options. 


Run a Program Directly 

Open the directory CP-examples in your file manager. If the JRE is installed properly 
you can start any one of the programs by simply clicking onto it. Under Linux, you 
can alternatively start it in a console window with e.g. 

java -jar CPexample.jar 

Figure A.2 shows a screenshot from computer exercise 23.4 (ladder model for expo¬ 
nential decay). 


Open a Program with the Netbeans Environment 

If you have the netbeans environment installed, you can import any of the pro¬ 
grams as a separate project by opening the corresponding folder in the directory 
CP-examples/NBprojects/. You may have a look at the source code and compile and 
run it 
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qmladderl? - NetBeans IDE 8.0.2 

£lle Edit View ftavigat Sourc« Refecto £un Debu^ £rofil< Tean Iools J(findov ijelp 


& e £i 


kiry ^ _j |<default config> ▼ ' 


^ NewJFramejava x NewClass java x 


Source Design History 
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317 
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320 

321 

323 

324 


»* * * iP ^ 2121 ® B 0 ® 


QE 


void rk4( double t) 

{ int i; 

grad(RealC, Image, RealFl, ImagFl, t); 

advance(RealC, Image, RealFl, ImagFl,RealC2, ImagC2, tstep*0.5); 
grad(RealC2, ImagC2, RealF2, ImagF2, t+tstep*0.5); 
advance (RealC,Image,RealF2,ImagF2, RealC3, ImagC3, tstep*0.5); 
grad(RealC3, Image3,RealF3,ImagF3, t+tstep*0.5); 
advance(RealC, Image,RealF3,ImagF3, RealC4, ImagC4, tstep) ; 
grad(RealC4, ImagC4, RealF4, ImagF4, t+tstep); 

for(i=0; i<nstates; i++) 

{ 

RealC[i]=RealC[i]+tstep*(RealFl[i]+2.0*RealF2[i]+2.0*RealF3[i]+RealF 
Image [i]= Image [ij+tstep*(ImagFl[i]+2.0*ImagF2[i]+2.0*ImagF3[i]+ImagF 

} 

} 

public class nyPanel extends javax.swing.JPanel{ 

int iy0,iyl,i,ix0, ixl.il; 
void iterate() 

{ 

} 

public void paintComponent(Graphics g) 
double t; int iy2,itick; 

Graphics2D g2=(Graphics2D) g; 


c? Usages m Output 


Q>i 


Fig. A.l Screenshot of the source code 
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Fig. A.2 Screenshot of computer experiment 23.4 




































Appendix B: Methods and Algorithms 


Purpose 

Method 

Comments 

Pages 

Interpolation 

Lagrange polynomial 

Explicit form, easy to evaluate 

19 

B ary centric Lagrange 
polynomial 

For evaluation at many points 

19 

Newton’s divided 
differences 

New points added easily 

21 

Neville method 

For evaluation at one point 

22 

Spline interpolation 

Smoother, less oscillatory 

22 

Rational interpolation 

Smoother, less oscillatory, 
often less coefficients necessary 

28, 32 

Pade approximation 

Often better than Taylor series 

29 

B ary centric rational 
interpolation 

Easy to evaluate 

30 

Rational interpolation 
without poles 

Alternative to splines, 
analytical 

34 

Multivariate 

interpolation 

Multidimensional 

35 

Trigonometric 

interpolation 

Periodic functions 

132 

Differentiation 

One-sided difference 
quotient 

Low error order 

39 

Central difference 
quotient 

Higher error order 

41 

Extrapolation 

High accuracy 

41 

Higher derivatives 

Finite difference methods 

43 

Partial derivatives 

Finite difference methods 

45 
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Purpose 

Method 

Comments 

Pages 

Integration 

Newton-Cotes formulas 

Equally spaced points 

49 

Trapezoidal rule 

Simple, closed interval 

49 

Midpoint rule 

Simple, open interval 

50 

Simpson’s rule 

More accurate 

49 

Composite 

Newton-Cotes rules 

For larger intervals 

50 

Extrapolation 

(Romberg) 

High accuracy 

51 

Clenshaw-Curtis 

expressions 

Suitable for adaptive and 

multidimensional 

quadrature 

53 

Gaussian integration 

High accuracy if 
polynomial approximation 
possible 

53 

Monte Carlo integration 

High dimensional integrals 

202 

Linear equations 

Gaussian elimination 
(LU reduction) 

Standard method for linear 
equations and matrix 
inversion 

64 

QR decomposition 

Numerically more stable 

69 

Iterative solution 

Large sparse systems 

78 

Richardson iteration 

Simplest iterative method 

79 

Jacobi relaxation 

Iterative matrix-splitting 
method, converges for 
diagonally dominant 
matrices, parallel 
computation possible 

80 

Gauss-Seidel relaxation 

Iterative matrix-splitting 
method, converges for 
symmetric positive definite 
or diagonal dominant 
matrices, no extra storage 

81 

Chessboard (black-red) 

Two independent subgrids, 
especially for Poisson 
equation 

402 

Damping and 

Successive 

over-relaxation 

Speeds up convergence for 
proper relaxation parameter 

81 
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Purpose 

Method 

Comments 

Pages 


Multigrid method 

Fast convergence but more 
complicated 

402 

Conjugate gradients 
method (CG) 

Krylov space method for 
symmetric positive definite 
matrices, preconditioning often 
necessary 

86 

General minimum 

residual method 
(GMRES) 

Krylov space method for 
nonsymmetric systems 

89 

special LU decomposition 

Tridiagonal linear equations 

75 

Sherman-Morrison 

formula 

Cyclic tridiagonal systems 

77 

Root finding 

Bisection 

Reliable but slow continuous 

functions 

98 

Regula falsi (false 
position) 

Speed and robustness between 
bisection and interpolation 

99 

Newton-Raphson 

Continuous derivative necessary, 
converges fast if starting point is 
close to a root 

100 

Interpolation (secant) 

No derivative necessary, but 
slower than Newton 

101 

Inverse interpolation 

Mainly used by combined 
methods 

102 

Dekker’s combined 

method 

Combination of bisection and 

secant method 

106 

Brent’s combined method 

Combination of bisection, secant, 
and quadratic inverse 
interpolation methods, very 
popular 

107 

Chandrupatla’s combined 
method 

Uses quadratic interpolation 
whenever possible, faster than 
Brent’s method, especially for 
higher order roots 

109 

Multidimensional root 
finding 

Newton-Raphson 

Needs full Hessian 

124 

Quasi-Newton (Broyden) 

Hessian not needed, no matrix 
inversion 

125 

Function Minimization 

Ternary search 

No gradient needed, very simple, 
for unimodal functions 

115 

Golden section search 
(Brent) 

Faster than ternary search but 
more complicated 

116 
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Purpose 

Method 

Comments 

Pages 

Multidimensional 

minimization 

Steepest descent 

Simple but slow 

122 

Conjugate gradients 

Faster than steepest descent 

124 

Newton-Raphson 

Fast, if starting point close to 
minimum, needs full Hessian 

124 

Quasi-Newton (BFGS, 
DFP) 

Hessian not needed, very popular 

125 

Fourier transformation 

GortzePs algorithm 

Efficient if only some Fourier 
components are needed 

136 

Fast Fourier transform 

Much faster than direct discrete 

Fourier transform 

138 

Time-Frequency 

Analysis 

Short Time Fourier 
Transform (STFT) 

Constant resolution for all 
frequencies, often used for audio 
signals 

145 

Gabor transform 

STFT with Gaussian window 
represents signal by elementary 
signals localized in time and 
frequency 

156 

Discrete STFT 

Reduced redundancy, still 
invertible 

153 

Continuous Wavelet 

transform 

Constant relative frequency 
resolution, better time resolution 
for high frequencies,very time 
consuming convolution integral 

158 

Discrete Wavelet 

Transform 

Uses orthogonal or biorthogonal 
wavelets, fast scalar product 


Multiresolution analysis 

Represents a signal by a basic 
approximation and a series of 
details with increasing resolution 

164 

Fast wavelet transform 

Recursive filtering, very fast 

178 

Random numbers 

Linear congruent 
mapping (LC) 

Simple pseudo-random number 
generator 

197 

Xorshift 

Fast, maximum possible period 

197 

Multiply with carry 
(MWC) 

Similar to LC but uses a varying 
carry 

198 

Complementary multiply 
with carry (CMWC) 

Improves MWC, passes many 
tests 

199 
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Purpose 

Method 

Comments 

Pages 


RN with given distribution 

Inverse of cumulative distribution 

function needed 

199 

Random points on unit 
sphere 

Random directions 

200 

Gaussian RN (Box-Muller) 

Gaussian random numbers 

201 

Thermodynamic 

average 

Simple sampling 

Inefficient 

206 

Importance sampling 

Samples preferentially important 
configurations 

207 

Metropolis algorithm 

Generates configurations 
according to a canonical 
distribution 

207 

Eigenvalue problems 

Direct solution 

Only for very small dimension 

214 

Tridiagonal matrices 

Explicit solutions for some 
special tridiagonal matrices 

217 

Jacobi 

Simple but not very efficient 

214 

Power iteration 

Finds dominant eigenvector 

225 

QL and QR 

Efficient power iteration method 
for not too large matrices, 
especially in combination with 
tridiagonalization by 

Householder transformations 

228 

Lanczos 

Iterative method for very large 
matrices or if only a few 
eigenvalues are needed 

230 

Singular value 
decomposition (SVD) 

Generalization for arbitrary 
matrices 

242 

Data fitting 

Least square fit 

Fit a model function to a set of 

data 

236 

Linear least square fit with 
normal equations 

Simple but less accurate 

237 

Linear fit with 
orthogonalisation 

Better numerical stability 

239 

Linear fit with SVD 

Expensive but more reliable, also 
for rank deficient matrices 

248 

Low rank matrix 
approximation 

Data compression, total linear 
least squares 

245 
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Purpose 

Method 

Comments 

Pages 

Discretization 

Method of lines 

Continuous time, discretized 
space 

261 

Eigenvector expansion 



Finite differences 

Simplest discretization, uniform 
grids 

259 

Finite volumes 

Partial differential equations with 
a divergence term (conservation 
laws), flux conservative, allows 
unstructured meshes and 

discontinuous material 
parameters 

265 

Finite elements 

Very flexible and general 
discretization method but also 
more complicated 

277 

Spectral methods 

Expansion with global basis 
functions, mostly polynomials 
and Fourier sums, less expensive 
than finite elements but not as 

accurate for discontinuous 
material parameters and 
complicated geometries 

273 

Dual grid 

For finite volumes 

265, 409 

Weighted residuals 

General method to determine the 
expansion coefficients 

270 

Point collocation 

Simplest criterion, often used for 
nonlinear problems and spectral 
methods 

271 

Sub-domains 

More general than finite volumes 

271 

Least square 

Popular for computational fluid 
dynamics and electrodynamics 

272 

Galerkin 

Most widely used criterion, leads 
often to symmetric matrices 

273 

Fourier pseudo-spectral 
method 

Very useful whenever a Laplacian 
is involved, reduces dispersion 

273 

Boundary elements 

If the Green’s function is 

available 

286 
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Purpose 

Method 

Comments 

Pages 

Time evolution 

Explicit forward Euler 

Low error order and unstable, 
mainly used as predictor step 

292 

Implicit backward Euler 

Low error order but stable, used for 
stiff problems and as corrector step 

295 

Improved Euler (Heun, 
predictor-corrector) 

Higher error order 

296 

Nordsieck 

predictor-corrector 

Implicit method, has been used for 
molecular dynamics 

298 

Gear predictor-corrector 

Optimized for molecular dynamics 

300 

Explicit Runge Kutta 
(2nd, 3rd, 4th) 

General and robust methods, easy 
step size and quality control 

301 

Extrapolation 

(Gragg-Bulirsch-Stoer) 

Very accurate and very slow 

305 

Explicit Adams-Bashforth 

High error order but not 
self-starting, for smooth functions, 
can be used as predictor 

306 

Implicit Adams-Moulton 

Better stability than explicit 
method, can be used as corrector 

306 

Backward differentiation 
(Gear) 

Implicit, especially for stiff 
problems 

307 

Linear multistep 
predictor-corrector 

General class, includes 
Adams-Bashforth-Moulton and 

Gear methods 

309 

Verlet integration 

Symplectic, time reversible, for 
molecular dynamics 

310 

Position Verlet 

Less popular 

312 

Velocity Verlet 

Often used 

313 

Stoermer-Verlet 

If velocities are not needed 

313 

Beeman’s method 

Velocities more accurate than for 

Stoermer-Verlet 

315 

Leapfrog 

Simple but two different grids 

317,317,471 

Crank-Nicolson 

Implicit, stable, diffusion and 
Schroedinger equation 

486, 474 

FTBS, Lax-Friedrich 

simple methods for advection 

434, 436 

Lax-Wendroff 

Hyperbolic differential equations 

472 

Taylor-Galerkin 

Lax-Wendroff 

highly accurate for advection 

449 

Two-step 

Differential equation with second 
order time derivative 

464 
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Purpose 

Method 

Comments 

Pages 


Reduction to a first order 
equation 

Derivatives treated as additional 

variables 

467 

Two-variable 

Transforms wave equation into a 
system of two first order 
equations 

470 

Split operator 

Approximates an operator by a 
product 

490,311, 

533 

Unitary time evolution 

Rational approximation 

Implicit,unitary 

526 

Second order differencing 

Explicit, not exactly unitary 

530 

Split operator Fourier 

Low dispersion, needs fast 

Fourier transformation 

533 

Real space product formula 

Fast but less accurate, useful for 
wavepackets in coupled states 

534 

Rotation 

Reorthogonalization 

Restore orthogonality of rotation 
matrix 

293 

Quaternions 

Optimum parametrization of the 
rotation matrix 

343 

Euler angles 

Numerical singularities 

343 

Explicit method 

Low accuracy, 
reorthogonalization needed 

335 

Implicit method 

Higher accuracy, orthogonal 
transformation 

338 

Molecular dynamics 

Force field gradients 

Needed for molecular dynamics 

361 

Normal mode analysis 

Small amplitude motion around 
an equilibrium 

364 

Behrendsen thermostat 

Simple method to control 
temperature 

371 

Langevin dynamics 

Brownian motion 

395 

Many body quantum 
systems 

Variational Quantum 
Monte-Carlo method 
(VQMC) 

Calculates energy for non 
separable trial wavefunctions 

205, 577 
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Normal modes, 364 

Nullclines, 508 
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Quasi-Newton methods, 113, 125 
Quaternion, 343, 345, 346 
Qubit, 569 

Qubit manipulation, 569 


R 

Rabi oscillations, 544 
Random motion, 395 
Random numbers, 187, 196, 199 
Random points, 200 
Random walk, 385, 397 
Rational approximation, 526 
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Taylor-Galerkin scheme, 450, 451 
Taylor series method, 298 
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Wave equation, 458 
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